I invite you to join my paid membership list to read this writing and 150+ high-quality data engineering articles:

Upgrade subscription

• If that price isn’t affordable for you, check this DISCOUNT

• If you’re a student with an education email, use this DISCOUNT

• You can also claim this post for free (one post only).

• Or take the 7-day trial to get a feel for what you’ll be reading.

Intro

In any database, a catalog is a central repository that stores metadata for all database objects, such as tables, columns, views, users, and relationships, acting as a directory to help the system find, understand, and manage data.

In the lakehouse world, there is also a concept of a catalog.

It is the central metadata layer, serving as a unified directory to discover, govern, and manage data across diverse sources (such as data lakes and warehouses) by tracking tables, schemas, and access rules, enabling analytic engines and AI models to access data without moving it.

Databricks, known as the vendor that publicly introduced the concept of lakehouse to the world, has built and provided a robust catalog service on top of their offering.

It’s called Unity Catalog. It has served as Databricks’s proprietary product for a long time before the vendor decided to open-source it last year.

In this week’s article, I share my insights on the Unity Catalog after reading the Databricks paper, Unity Catalog: Open and Universal Governance for the Lakehouse and Beyond.

Problems Unity Catalog was trying to solve

Following the paper, UC has been serving the 9000 Databricks customers since 2021 with some interesting numbers: 100 million tables under management, 400,000 machine learning (ML) models, and 60,000 API calls per second (yeah, you read it right, per second). A good piece of software always starts with the user's problems in mind. Here are challenges that Unity Catalog was designed to address:

• Uniform access control: A secure approach to let Lakehouse’s users access the assets (e.g., the tables or the ML models) in both ways: via catalog or via a direct cloud storage path.

  

• Support for diverse asset types: The catalog must support a wide range of assets, adapting to the user’s needs. In addition to tables, users also need to govern ML models.

  

• External access: Users require the data sharing capability done via the catalog; the key is that it must avoid data copying (which might increase storage cost and increase complexity when you must ensure data synchronization between the two copies).

  

• Discovery support: The assets must be discoverable and understandable. Their lifecycle and lineage must also be transparent

  

• Performance: Users don’t want it to be slow, even though it only involves metadata-related operations.

  

How does Unity Catalog fit into the Databricks lakehouse architecture?

The architecture consists of three key components:

• The storage: As you might know, the lakehouse paradigm separates the storage and compute for flexibility and interoperability. The storage could be object storage services from famous vendors like AWS or Google. In addition to the physical data stored in Parquet, there is a metadata layer on top of it that provides the table abstraction. Delta Lake and Iceberg are used for this purpose.

• The Databricks runtimes: The Databricks’ core execution engine, which is a fork of Apache Spark that provides the same interface but has enhancements for reliability and performance. Databricks also built the Photon engine, a library that integrates closely with DBR to enhance Spark analytical workloads further. The engine acts as a new set of physical operators inside the DBR. The query plan can use these operators as any other Spark query. Databricks’ customers can continue to run their workloads without any changes and still benefit from Photon.

Here is my previous article about How is Databricks’ Spark different from Open-Source Spark?

• The Unity Catalog service: And finally, today’s primary focus, the Unity Catalog service. It’s a multi-tenant service that provides all UC functionality and APIs. The clients (in most cases, the query engines) use this service to deliver what the user needs.

I invite you to join my paid membership list to read this writing and 150+ high-quality data engineering articles:

Upgrade subscription

• If that price isn’t affordable for you, check this DISCOUNT

• If you’re a student with an education email, use this DISCOUNT

• You can also claim this post for free (one post only).

• Or take the 7-day trial to get a feel for what you’ll be reading.

The SQL query’s journey

After having a glimpse of Databricks lakehouse architecture, we will understand a typical SQL query journey:

• Upon DBR receiving the SQL query from the user, it parses the query to extract the related data asset references, such as the table or view names.

• DBR then sends a REST request to the Unity Catalog (UC) service to verify whether the user has the required permission on the data assets. Then the UC will return the assets’ metadata, such as column definitions and constraints.

• The DBR will use the return metadata for the planning process.

• When the DBR needs to access the data in the object storage, it issues another request to fetch a temporary credential.

• The DBR can now access the data using the returned credentials.

• The query can now be processed, from scanning data to filtering, joining, and aggregating. The result will be sent back to the user.

The designs

After understanding the high-level of the Unity Catalog, we will dive into its system design to see how Databricks resolved the challenges discussed at the beginning of the article.

The disaggregation of the catalog and engine

First, Databricks decides that the Unity Catalog will be an independent service, not tied to any specific query engine. This helps two things:

• Security: The Unity Catalog can act as a gatekeeper; only those with authorized permission can access assets managed by the catalog.

• Interoperability (the spirit of the lakehouse architecture): Different query engines can work with data managed by the catalog. The query engine accesses the metadata via the REST API, rather than replicating the metadata to each engine.

Diverse Asset Types management

As mentioned, in addition to tables, users need the Unity Catalog to manage various data asset types.

The foundation of this ability lies in the entity-relationship (ER) data model, which backs all metadata operations.

The model exposes methods for tasks such as ID- or name-based lookup, parent-child relationship mapping, privilege grant management, and state management for resource provisioning and cleanup. Developers can extend the model for a specific asset type. This model is materialized in a relational database.

On top of the model is the adapter layer. The primary responsibility of this layer is to offer integration points for different asset types with various cloud providers.

A developer can register an asset type in the catalog by adding a manifest to UC’s asset types registry. The manifest includes information such as its location in the model hierarchy, the support operations and privileges, the rules associated with each operation, and how the catalog will manage the asset lifecycle.

In this way, multiple asset types can be added to the Catalog.

Right on top of the adapter is the core features layer, which provides methods like namespace, lifecycle management, access control, and audit logging.

This layer exposes APIs for metadata management and credential requests.

Access control

Databricks wants to ensure that data access is secure, regardless of how it is accessed: via the catalog or directly via the object storage path.

First, Databricks protects all the assets at two distinct levels: metadata and data. Metadata security is handled directly through UC’s REST APIs, where the system validates a user's permissions based on the specific operation being performed.

Rather than giving clients direct access to cloud storage, administrators grant storage permissions exclusively to the Unity Catalog service. When a client needs to read or write data, it requests temporary credentials from the UC.

If the engine needs to access the data via the object storage path, the UC will resolve the path to the asset identifier, check whether the client has sufficient permissions, and finally return a temporary credential.

For more fine-grained access control, such as row or column security, the UC works with the engine to enforce a two-level access control. In addition to the mechanism above to protect both the metadata and the data of a table, the engine is now responsible for row and column security.

A discoverable catalog

Recalled from the “Diverse Asset Types management” section, Unity Catalog is designed as a layered architecture, with the bottom layer the entity-relation model stored in the relational database, followed by the core feature layer and the APIs layer.

On top of the APIs layer is a layer with background functions, such as discovery, search, and data lineage. The service must be notified and updated whenever the metadata is changed.

Whenever changes occur, the core service sends change events so the background functions can consume them to update the indexes or lineage graphs. Databricks adopts this event-driven architecture to ensure that background functions stay up to date with the core service.

The separation between the core service and the background functions ensures both can be scaled independently, and the failure of a service won’t impact the remaining services.

The core service also provides access control for background functions, ensuring that only authorized clients can discover the assets.

Performance

As UC is the entry point for all the analytical workload in Databricks, its access latencies directly impact the user experience. Here are some optimizations that Databricks implements for the UC:

• Batching multiple asset requests.

  

• Caching immutable metadata (e.g., temporary credentials) at both the UC service and the query engines.

• For mutable metadata (e.g., table’s name, columns, permissions…), Databricks implements a write-through cache for the relational database that backs the metadata. The relational database’s data will be sharded across nodes; each node is responsible for its assigned subset of data in both the database’s storage and the cache.

  

• To prevent the cache from growing indefinitely. Databricks implements two strategies for cache eviction. First, there are standard algorithms, such as LRU, that discard unpopular cached entities. Second, to cap the number of cached versions of popular entities, Databricks enforces a timeout for all UC API calls. They assume that when a request to an asset populates a new version of the asset into the cache, the existing cached versions will be used for processing requests for at most the timeout period.

In a write-through cache, data is written simultaneously to the cache and the primary storage system. This approach ensures that the cache always contains the most recent data. — from What is Read-Through vs Write-Through Cache? by Design Guru

• UC uses database versions to provide snapshot isolation and serializable isolation.

The idea of snapshot isolation (SI) is that each read transaction reads a consistent snapshot of the database. The transaction will only see all changes committed before it starts.

Serializable is the strongest isolation level: although transactions can run in parallel, their effects are the same as if they were run serially (one at a time). There are several approaches to implement serializability, including serialized snapshot isolation (SSI).

SSI is still based on the SI; the readings are still served with consistent snapshots. However, SSI has mechanisms to detect conflicts between writes. A common approach is to detect whether the reading snapshot is stale.

For a given transaction, SI provides it with a consistent snapshot and ignores changes from ongoing or new transactions. With SSI, the database can check if any ignored changes were committed during the transaction. If so, the database will abort the transaction if it’s not a read-only transaction — from ACID For Data Engineers by Vu Trinh.

Outro

In this article, I’ve shared my insights after reading the paper Unity Catalog: Open and Universal Governance for the Lakehouse and Beyond. From the problems that Unity Catalog is trying to solve, how it fits into the Databricks lakehouse architecture, the query life cycle with the Catalog as the entry point, and finally, the system designs that made it a reliable, secure, and performant lakehouse catalog

Reference

[1] Databricks, Unity Catalog: Open and Universal Governance for the Lakehouse and Beyond, 2025

I will publish a paid article every Tuesday. I wrote these with one goal in mind: to offer my readers, whether they are feeling overwhelmed when beginning the journey or seeking a deeper understanding of the field, 15 minutes of practical lessons and insights on nearly everything related to data engineering.

Upgrade subscription

Intro

As Joe Reis and Matt Housley once said in the infamous book, Fundamentals of Data Engineering:

A data engineer manages the data engineering lifecycle, starting with extracting data from source systems and concluding with serving data for specific use cases.

The data serving is the primary interface through which we provide our service to end users (e.g., data analysts, data scientists, business stakeholders). No matter how well we store, process, and manage the data, if users cannot access or use it reliably, we have failed.

Today, I want to discuss one of the hottest methods for serving data in the era of AI: natural language to SQL. We will first understand why text-to-SQL is receiving a lot of attention recently, what its challenges are, and then attempt to find a solution that addresses them.

Why Text-to-SQL

In the past, if business users wanted to gain insight from the data, they had to communicate with the IT department so that these technical experts could assist them.

The business intelligence tools have evolved since then. More functionalities, a shinier UI, the ability to connect to more systems, and most importantly, more friendly to non-technical users.

From asking the technical team for help, business users can now build their own charts or create reports with the assistance of modern business intelligence tools, which allow them to drag-and-drop the data fields they want.

But it seems like that’s not enough. The rise of AI chat interfaces like ChatGPT or Gemini makes people realize that “oh, using natural language is even more productive compared to the visual drag-and-drop.“ BI tools on the market are starting to integrate the ability to answer human questions with the help of AI models.

The key is to enable the AI models to translate user input into SQL queries. Then, the tool will send the SQL to the database and create a chart/report based on the results.

Instead of choosing the `total_sales` and `country` fields, a simple text, “Show me the total sales breakdown by country in the last month,” is more intuitive for the users. Integrating with AI makes a solution more compelling.

Challenges of Text-to-SQL

I refer to the paper “A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going?” for this section.

Instructing AI models to accept natural language input and output a reliable SQL query is not easy to achieve. To better understand the challenges, let’s first revisit some steps that humans take to write SQL:

• We begin with the business question, the natural language query: for example, all countries with sales greater than 2,000 on Independence Day.

  

• In our brain, we identify the entities: the countries, the sales, the context: June, and the condition: sales greater than 2,000.

  

• We find the relevant tables, columns, and records by examining the database schema. The human interpretation is essential here, which kind of sales (assume the company has more than one product), and what date is Independence Day? (This varies in countries.) This step may require us to revisit the business users to request additional information.

  

• Then, we write SQL based on our understanding. We Select, Join, Group By, Where…

We, humans, despite knowing what we are trying to do, still have some challenging problems while handling the “text-to-SQL “ process: the uncertainty of the natural language, the database’s complexity, and the translation from the “flexible” natural language queries to the “strict” SQL queries.

Natural language uncertainty

We use natural language from the day we learn to say our first words, such as “mama” or “papa“. We practice it every day, and the way we communicate depends significantly on who we are, how we grew up, and how we perceive the world.

It’s normal for us to say a thing, and others understand it in different ways. This is called ambiguity. It could happen when a single word has multiple meanings, …

…or a sentence can be parsed in various ways.

The uncertainty also stemmed from under-specification, which occurs when expressions lack sufficient detail or context to convey their intended meanings. For example, Independence Day in Vietnam is different from Independence Day in the United States of America.

We can ask others, observe around, or leverage our experience and understanding to resolve the ambiguity. Meanwhile, the AI models might only have a natural language query.

The database’s complexity

It’s common for us, data engineers, to handle messy data systems. Lack of robust data modeling, complex relationships between tables, ambiguous columns, or more than one way to calculate a metric.

Let’s confess here, it is tough for us to do the right thing the first time with this data system. We might run around the companies to ask for more clarification, cause some bugs, and create some weird reports before learning how to do it right. An AI model, somewhere on the internet, knows nothing about your company’s data system. How could we expect it to do better than us?

Text-to-SQL Translation

For the machine to understand, our Python or Java code must be translated into low-level machine language. This is a complex task, but at a high level, things are straightforward, as each language has a kind of dictionary to facilitate a one-to-one mapping between programming language code and machine code.

However, converting text to SQL is more challenging than that, as it typically involves a one-to-many mapping between the input natural language query ←→ database entities and natural language query ←→ SQL query.

Natural language is flexible, whereas SQL queries must adhere to a strict syntax. Even SQL queries could have different syntax depending on the standard and the database implementation.

We require not only that the queries be executable, but also that they be readable, optimized, and reliable. Placing this responsibility on the AI models seems to overwhelm them, given that they may return low-performance queries, hard-to-debug ones, inaccurate results, or multiple SQL queries for the same prompt.

So, is there a way for us to deal with these problems?

It turns out that there is a promising approach.

In the paper “A benchmark to understand the role of knowledge graphs on large language models’ accuracy for question answering on enterprise SQL databases”, the author created a robust benchmark series of questions with different levels of complexity using a standardized insurance dataset. They asked ChatGPT to answer the questions in two ways:

• Generate the SQL directly

• Generate the SQL with the help of a knowledge graph

They observed that leveraging the knowledge graph indeed helps improve the accuracy of results:

Essentially, a knowledge graph is a structured way to represent knowledge about entities and their relationships, utilizing a graph-based data model. There is a popular solution that offers the same benefit.

Yes, it is the semantic layer

As a company’s business expands, the volume and variety of data increase; more decisions need to be made, more data must be stored, and more source data must be captured. Despite how well we prepare, data users might struggle to understand what they need to use the data effectively. We need a better abstraction layer that can lower the barrier for people.

The semantic layer is an abstraction layer that sits between the underlying data (e.g., data warehouses) and end-user applications (e.g., BI tools, data applications, or business users). From a high level, a semantic layer solution requires us to map business-friendly concepts to underlying data assets and specify the relationships between them.

Thanks to that, the layer acts as a translator between the data and its users. It abstracts all the complexity to ensure that only understandable and business-friendly concepts are presented to users.

Semantic layer’s role in Text-to-SQL tasks

Recall that ambiguity and database complexity affect the accuracy of the text-to-SQL system. With the help of the semantic layer, the Text-to-SQL output could be more reliable:

• AI models don’t need to understand the database complexity anymore, as all the information they require is baked into the semantic layer, from the tables needed to the right way to join them. In other words, an AI model is enriched with context through the semantic layer.

  

• When a user requests “total sales,” the AI does not need to infer or guess the logic; it can simply reference the predefined “Total Sales” metric in the semantic layer, which already contains the calculation. This limits the ambiguity.

A real-world example

The semantic layer has emerged lately, given its ability to abstract the complexity of the underlying data systems. As discussed, this is not only a benefit to business users but also to the AI models. The layer is an indispensable part of modern BI tools, such as Tableau, Looker, and Power BI, as well as an interesting solution called Holistics.

Established in 2015, the platform enables self-service data access for the entire organization. Compared to other BI tools, if users want to extract insight on Holistics, they must define their mapping between business concepts and the underlying tables via the semantic layer. Only after that, users can start presenting and organizing data using concepts exposed from the semantic layer.

To work with the semantic layer, Holistics introduces the concept of “model“, which is an abstract representation on top of a table/query. A model should have the source (a physical table or a SQL query), the dimensions and measures, and the relationships to other models. Holistics uses relationships for constructing the join.

With Holistic’s vision of the semantic layer from the beginning, it would be easier for them to develop the text-to-SQL feature. They’ve tried several approaches, including letting the AI models offload the generation of SQL to the semantic layer by translating the user’s natural language input to a format that the semantic layer could understand, such as a JSON payload.

By doing it this way, the text-to-SQL process can become even more reliable, as the SQL queries are now controlled by the semantic layer, which is designed to generate output queries based on well-tested logic and predefined entities within the semantic layer. Compared to the fact that the AI model has to guess, this way is more reliable.

Even with the semantic layer, it might not be enough for text-to-SQL

Although relying entirely on the semantic layer could be beneficial, this approach may be limited by the fact that the input format, such as JSON, doesn’t provide users with the necessary flexibility in cases of complex analytics requirements.

For example, with the pseudo-format like this:

{ "metrics": ["total_sales"], "dimensions": ["country"]}

It serves well for simple questions. However, the key-value formats could cause users trouble when expressing queries that require more advanced techniques, such as nested aggregation or period-over-period comparison.

So, letting the AI model generate the SQL directly is less reliable, but interacting via the semantic layer with the intermediate format is less flexible. What do we do?

Holistics chooses to let the AI model generate the queries, but in a more reliable and controllable way. The model still leverages the help of the semantic layer for the business context and understanding; however, it has been trained to generate a new kind of query language instead of SQL. They call this AQL. s. Let’s delve into this language before moving on.

The AQL language

When the platform was first built, the creator behind Holistics had already developed a proprietary language for analytics, known as AQL. This language is designed to leverage the defined semantic layer, allowing us to query data at a higher level of abstraction.

AQL treats metrics as first-class citizens, making metric definition composable and reusable. This differs from SQL, where everything is a query. If you want to reuse a piece of metrics, you must save the query that calculates it somewhere, such as in a CTE, a view, or a table. When adjusting the metric logic, you must modify the query.

AQL queries are written using business concepts (dimensions and measures) defined in the semantic layer, not raw table and column names. A user can ask for `total_revenue` by `user_country` without having to write the complex JOIN statements. This abstraction simplifies query writing and drastically improves the readability and maintainability of analytics code.

Additionally, AQL introduces the pipe operator |, which takes the result of the expression on its left and uses it as the input for the function on its right. This creates a clear, sequential, top-to-bottom flow of logic.

Users express their metrics using AQL; then, Holistics converts them to SQL queries and executes them on the defined database.

The solution

Back to Holistics, the way they build the text-to-SQL will look like this: they trained their AI models to accept natural language input and output the AQL queries with the help of the semantic layer. The AQL query is then converted to a SQL query.

The outcomes are AI-generated queries that are fundamentally more verifiable, reliable, and governed than those produced by systems that attempt direct text-to-SQL translation:

• Verifiable & Readable: Because AQL is a high-level language that operates on business logic, the queries it generates are far more compact and intuitive than raw SQL. A user can look at a piped AQL query and immediately understand the logical steps the AI is taking and ensure that AI really gets what the intent of the question is about

  

  • This human-readability is critical for verification; it allows the model trainer or the end users to understand what the AI is doing. This is an improvement compared to spending time reading messy SQL queries.

  • The high level abstraction AQL provides reduces risks of errors and hallucination as compared to the risk of AI errors from interpreting and using low level SQL queries from scratch.

  • Because the AQL-to-SQL conversion is managed by Holistics’ well-tested system, the generated SQL query is guaranteed once the AQL is correct.

• Reliable: By abstracting away the most error-prone aspects of query generation—such as dialect-specific syntax, complex join logic, and the formulas for advanced analytics—the system significantly increases its reliability.

  

  • The AI’s task is simplified to mapping intent to predefined metrics and dimensions in AQL. This leads to more accurate and dependable results.

• Governed: Because every AQL query must operate through the semantic layer, it automatically inherits the organization’s single source of truth for business definitions.

  

  • The AI won’t invent its metric calculations. Furthermore, access controls defined in the semantic layer are automatically enforced, ensuring that users can only query data to which they are authorized.

• Flexibility: AQL is designed to express complex metrics seamlessly, including AI; the capability of a text-to-SQL system will not be limited to simple queries only due to the limitation of the intermediate format, such as JSON.

Outro

In this article, we first explore why extracting data insights using natural language is gaining increasing attention. Next, we examine the challenges of Text-to-SQL and find out that there is a promising solution to improve the accuracy with the help of the semantic layer.

Finally, we examine a real-life example: Holistics, which understands its solution to Text-to-SQL by leveraging semantic layers and its self-developed analytics language, AQL.

Thank you for reading this far. See you next time.

Reference

[1] Phuc Nguyen, The Ideal Semantic Layer and Metric-Centric Paradigm, 2023

[2] Tan Huynh, Metrics Deserve Better Composition Than What SQL Allows, 2024

[3] Holistics AI Architecture

[4] Holistics Official Documentation

[5] Justin Heinze, History of Business Intelligence, 2020

To celebrate Lunar New Year (the true New Year holiday in Vietnam), I’m offering 50% off the annual subscription. The offer ends soon; grab it now to get full access to nearly 200 high-quality data engineering articles.

50% off the annual subscription

Intro

Orchestrating the data pipeline is as crucial as its task logic and performance. Luckily, we don’t have to do that from scratch; many available tools can help us.

Among them, Airflow appears to be the dominant solution, thanks to its openness and active community. However, as data engineers, writing DAG files was not enough; we needed to understand the underlying concepts to operate the tool confidently.

This article will take a closer look at one of the most important aspects of Airflow: the mechanism by which tasks are run, Airflow’s executor.

We first briefly revisit Airflow, and then we explore the executor and its available options.

The History

Apache Airflow was created in 2014 at Airbnb when the company was dealing with massive and increasingly complex data workflows. At the time, existing orchestration tools were either too rigid, lacked scalability, or couldn’t accommodate the dynamic nature of data pipelines. To address this challenge, Maxime Beauchemin, a data engineer at Airbnb, spearheaded the creation of Airflow.

Airflow quickly gained traction and, in 2016, joined the Apache Software Foundation, becoming an open-source project with a robust and growing community.

If you've joined a new company these days, you're likely to work with Airflow.

Overview

Orchestrating a complete data pipeline presents numerous challenges. When should we schedule the data retrieval from a third-party API? How do we effectively manage dependencies between the API call and the data processing job? What happens in the event of a failure? Can we observe it? If so, can we retry?

Apache Airflow simplifies this problem by allowing engineers to define workflows as code and automating their execution.

At its core, Airflow operates on the concept of Directed Acyclic Graphs (DAGs) to model workflows. It is essentially a roadmap for the workflow and contains two main components:

• Tasks (Nodes) are individual work units, such as running a query, copying data, executing a script, or calling an API.

• Dependencies (Edges): The relationships between tasks that define their execution order (e.g., preprocessing is executed only after retrieving data from a third-party API).

Airflow ensures tasks are executed sequentially or in parallel (based on their dependencies), automatically manages retries on failure (based on their retry configuration), and thoroughly logs task execution for monitoring and debugging purposes.

The Internals

There are several components inside Airflow:

1. Scheduler: The component responsible for parsing DAG files, scheduling tasks, and queuing them for execution based on their dependencies and schedules. The executor logic runs inside the scheduler.

2. Web Server provides the Airflow UI, allowing users to visualize workflows, monitor task execution, inspect logs, and trigger DAG runs.

3. Metadata Database: A central database that stores all metadata, including DAG definitions, task states, execution logs, and schedules. It’s essential for tracking the history of workflows.

4. DAG folders: It contains DAG files defined by users.

5. Workers: Components that execute the tasks assigned by the executor.

   The executor is our main dish today, and we will discuss it very soon

Workflow Between Components

The workflow between Airflow’s components can be broken down into the following steps:

1. DAG defining: The users define the DAG with desired tasks and logic, including when to begin running it and the scheduled interval.

2. DAG Parsing: The Scheduler scans the DAG directory, parses the DAG file, and loads them into the Metadata Database.

3. Scheduling: Based on the DAG definitions and schedule intervals, the Scheduler determines which tasks are ready for execution and queues them.

4. Task Execution: The Executor fetches the queued tasks and assigns them to available Workers. The Workers execute the tasks, and task states are updated in the Metadata Database.

5. Monitoring: The Web Server queries the Metadata Database and visualizes DAG runs, task statuses, and logs in real-time. Users can monitor task progress, inspect logs, or trigger manual DAG runs from the UI.

6. Retries and State Updates: If a task fails, the Scheduler ensures retries are handled according to the task configuration. The Executor updates task states in the database until all tasks are completed successfully or fail beyond retry limits.

This article is sponsored by Astronomer. At Astronomer, a team of just five data engineers manages 27,000 daily tasks powering 18+ data products. After a series of architectural changes, they reduced DAG failure rates by 81%. On August 21, they’re sharing exactly how they did it, and how you can apply the same strategies to make your own pipelines more reliable. You’ll learn how to:

• Use Airflow Asset scheduling to prevent upstream data issues

• Orchestrate cross-DAG dependencies with a Control DAG

• Set up centralized observability to monitor SLAs and debug faster

Register HERE

Deployment

Deploying Airflow ranges from running a lightweight local instance for testing and development to setting up a robust, scalable, and production-ready environment. Here's an overview of the deployment process:

On a single machine

Airflow can be deployed directly on a single machine (airflow standalone) for testing and development. This setup will initiate all the required components (scheduler, web server, and database) as separate processes on our machine.

Another way to deploy Airflow on a single machine is by separting each process into a separate container (via Docker or on a local Kubernetes cluster such as Minikube)

However, a single-machine deployment is insufficient when operating Airflow in production, which requires scalability, availability, and fault tolerance.

Distributed Deployment

Airflow can be deployed in a distributed architecture; components are deployed independently and redundantly; each element is live on a separate machine and can be optionally deployed in multiple instances on different machines. (e.g., scheduler and webserver are on two other machines, the scheduler can have three instances deployed on three machines)

This setup enables better load distribution, making it well-suited for handling large-scale workflows. The most common approach for deploying Airflow's distributed architecture that I observed is using Kubernetes.

If you use Airflow managed by a cloud vendor like AWS or Google, your Airflow environment is deployed on a Kubernetes cluster, and all the DAG files are stored in the object storage (S3 for AWS and GCS for Google Cloud)

Executors

Now, the main dish.

Executors in Airflow are responsible for determining where and how tasks are executed. Different executors offer varying levels of scalability, isolation, and complexity.

SequentialExecutor

Categorized as Local Executor, it is replaced by the LocalExecutor in Airflow 3

This executor runs tasks sequentially (one after another) within a single process on the same machine as the scheduler. This executor is most used for development and local testing. It's simple but unsuitable for production due to its lack of parallelism.

A critical operational detail is that this executor pauses the scheduler while a task is running. This characteristic is a significant concern for production environments, as it prevents the scheduler from continuously monitoring or queuing new tasks.

The SequentialExecutor is also unique in its ability to operate with SQLite as its database backend, a choice that aligns with its single-task nature due to SQLite's lack of support for multiple concurrent connections.

Pros

• Its greatest strength is simplicity, requiring no external dependencies or complex configurations.

Cons

• Can’t run tasks in parallel

LocalExecutor

Categorized as Local Executor

The LocalExecutor represents an advancement over the SequentialExecutor by introducing parallelism while maintaining a relatively simple setup on a single machine. Concurrency is achieved through multiple processes on a single machine. It is suitable for small—to medium-sized workflows that require concurrency but don't need distributed execution.

To support this parallelism, a robust database backend such as MySQL or PostgreSQL is required, as SQLite does not handle the multiple connections necessary for concurrent operations.

The LocalExecutor has modes:

• Unlimited Parallelism (parallelism == 0): In this mode, a new process is spawned for every task submitted. Upon task completion, the process terminates. This represents a direct, on-demand approach to task execution.

• Limited Parallelism (parallelism > 0): This is the more common configuration for a production environment. A fixed number of worker processes (equal to the parallelism) are pre-spawned at startup. These workers continuously pull tasks from the queue, remaining active throughout the executor's lifecycle.

When there are multiple Schedulers, each will run a local executor. This means tasks will be distributed across the Schedulers’ machines.

Pros

• The simplicity

• Can leverage multiple CPU cores on the host machine, leading to higher concurrency compared to the SequentialExecutor

Cons

• Limited by the resources (CPU, RAM, etc.) of the Scheduler machines. More task processing capability means adding more Scheduler machines.

CeleryExecutor

Categorized as Remote Executor

The CeleryExecutor allows us to enter distributed systems and horizontal scaling. It relies on Celery, a robust distributed task queue. Compared to the two above executors, the workers who run the task are separate from the scheduler machines.

The CeleryExecutor setup involves:

• A message broker (most commonly RabbitMQ or Redis)

• Celery workers

Celery workers are typically long-running processes that continuously run to pick up tasks, allowing more than one task to run concurrently on a worker. To scale the task-running capability, we add more machines that run Celery worker processes. Similar to the LocalExecutor, it requires a robust, non-SQLite database (e.g., MySQL or PostgreSQL)

Pros

• Decoupling the task running process from the Scheduler.

• Horizontal scaling by adding more machines that run the Celery workers.

Cons

• More components compared to the local executors → More maintenance overhead

• Noisy Neighbor: A heavy task could affect other functions on the shared machine that runs the Celery worker.

• Not so good resource utilization as Celery workers could stay idle: running a fixed number of Celery workers continuously can lead to underutilized resources when few tasks are running.

• The overhead of scaling worker machines.

KubernetesExecutor

Categorized as Containerized Executor

This executor is designed for cloud-native and containerized environments. This executor dynamically creates Kubernetes pods for each task. For me, this one provides the best resource isolation, scalability, and fault tolerance.

When the Airflow scheduler senses that a task is ready for execution, it requests a new pod from the Kubernetes API. This newly created pod then executes the task, reports its result back to the Airflow metadata database, and terminates upon task completion (users can choose to persist the pod for debugging later)

Pros

• Better resource utilization: resources are consumed only when tasks are actively running, leading to cost savings during idle periods.

• Better isolation: each task can have its own pod with configurable resources. Unlike the above executors, KubernetesExecutor allows for better Python dependencies, as different tasks (pods) can have different sets of dependencies.

Cons

• The cold start: The Kubernetes pod needs to be initiated (e.g., pull the Docker image, run) to run the task; it could take a while compared to the above executors before executing your tasks.

• Requires strong knowledge of containerization and Kubernetes, which potentially requires more resources to manage (e.g., SRE teams)

• Hard to test as it requires users to have a Kubernetes environment.

Multiple Executors

Until Airflow 2.10, an Airflow environment was limited to using a single executor for all its tasks. However, with the introduction of multiple executor support (starting with Airflow 2.10 and later), users can specify different executors for different tasks.

Most of the time, a single executor is sufficient. However, with diverse workloads, the “one size fits all" approach may not be effective. Multiple executors could help here. For example:

• Short, Numerous Tasks: Some DAGs might consist of very small tasks. A CeleryExecutor with pre-warmed workers can excel here due to low task startup latency. Small tasks also mean quick-returning resources, limiting the noisy neighbor problems.

• Long-Running, Resource-Intensive Tasks: Long-running tasks consume significant CPU/memory on Celery workers, which might lead to "noisy neighbor" problems. With KubernetesExecutor, each task gets its own isolated pod with precisely allocated resources.

Outro

In this article, we first revisit the fundamentals of Airflow and then explore its common executor options, ranging from local ones with simple setups to limiting concurrency with Sequential and Local executors.

We then move on to the distributed option with CeleryExecutor, and finally learn about the most isolated and scalable option with KubernetesExecutor. However, more power comes with more responsibility, as this last one requires strong knowledge of Kubernetes to operate smoothly.

We also learn that Airflow allows us to specify more than one executor in a single environment.

Thank you for reading this far. See you next time.

References

[1] Apache Airflow Official Documentation

[3] Airbnb Engineer, Airflow: a workflow management platform (2015)

To celebrate Lunar New Year (the true New Year holiday in Vietnam), I’m offering 50% off the annual subscription. The offer ends soon; grab it now to get full access to nearly 200 high-quality data engineering articles.

50% off the annual subscription

Intro

At the time of this writing, Apache Spark has been released in its fourth major version, which includes many improvements and innovations.

However, I believe its core and fundamentals won’t change soon.

I have written this article to help you establish a good baseline for learning and researching Spark. It distills everything I know about this infamous engine.

Note: This article contains illustrations with many details. I recommend reading it on a laptop or PC to get the full experience.

Overview

In 2004, Google released a paper introducing a programming paradigm called MapReduce to distribute the data processing to hundreds or thousands of machines.

In MapReduce, users have to explicitly define the Map and the Reduce functions:

• Map: It takes key/value pair inputs, processes them, and outputs intermediate key/value pairs. Then, all values of the same key will be grouped and passed to the Reduce tasks.

• Reduce: It receives intermediate values from Map tasks. It then merges the intermediate values from the same key using the defined logic (e.g., Count, Sum, ...)

To ensure fault tolerance (e.g., a worker dies during the process), MapReduce relies on disk to exchange intermediate data between data tasks.

Based on Google's paper, Yahoo released the open-sourced implementation of MapReduce, which soon became the go-to solution for distributed data processing. It rose and dominated, but it wouldn’t last long.

The strict Map and Reduce paradigm limits the flexibility, and the disk-based data exchange might not be suitable for use cases like machine learning or interactive queries.

UC Berkeley’s AMPLab saw a problem that needed to be solved. Although cluster computing had a lot of potential, they observed that the MapReduce implementation might not be efficient.

They created Apache Spark, a functional programming-based API to simplify multistep applications, and developed a new engine for efficient in-memory data sharing across computation steps.

Spark RDD

Unlike MapReduce, Spark relies heavily on in-memory processing. The creator introduced the Resilient Distributed Dataset (RDD) abstraction to manage Spark’s data in memory. No matter the abstraction you use, from dataset to dataframe, they are compiled into RDDs behind the scenes.

RDD represents an immutable, partitioned collection of records that can be operated on in parallel. Data inside RDD is stored in memory for as long as possible.

Why RDD immutable

You might wonder why Spark RDDs are immutable. Here are some of my notes:

• Concurrent Processing: Immutability keeps data consistent across multiple nodes and threads, avoiding complex synchronization and race conditions.

• Lineage and Fault Tolerance: Each transformation creates a new RDD, preserving the lineage and allowing Spark to recompute lost data reliably. Mutable RDDs would make this much harder.

• Functional Programming: RDDs follow principles that emphasize immutability, making handling failures easier and maintaining data integrity.

Properties

Each RDD in Spark has five key properties:

• List of Partitions: An RDD is divided into partitions, Spark's parallelism units. Each partition is a logical data subset and can be processed independently with different executors (more on executors later).

• Computation Function: A function determines how to compute the data for each partition.

• Dependencies: The RDD tracks its dependencies on other RDDs, which describe how it was created.

• Partitioner (Optional): For key-value RDDs, a partitioner specifies how the data is partitioned, such as using a hash partitioner.

• Preferred Locations (Optional): This property lists the preferred locations for computing each partition, such as the data block locations in the HDFS.

Lazy

When you define the RDD, its data is unavailable or transformed immediately until an action triggers the execution. This approach allows Spark to determine the most efficient way to execute the transformations. Speaking of transformation and action:

• Transformations, such as map or filter, define how the data should be transformed, but they don't execute until an action forces the computation. Because RDD is immutable, Spark creates a new RDD after applying the transformation.

• Actions are the commands that Spark runs to produce output or store data, thereby driving the actual execution of the transformations.

Fault Tolerance

Spark RDDs achieve fault tolerance through lineage.

As mentioned, Spark keeps track of each RDD’s dependencies on other RDDs, the series of transformations that created it.

Suppose any partition of an RDD is lost due to a node failure or other issues. Spark can reconstruct the lost data by reapplying the transformations to the original dataset described by the lineage.

This approach eliminates the need to replicate data across nodes or write data to disk (like MapReduce).

Architecture

A Spark application consists of:

• Driver: This JVM process manages the entire Spark application, from handling user input to distributing tasks to the executors.

• Cluster Manager: This component manages the cluster of machines running the Spark application. Spark can work with various cluster managers, including YARN, Apache Mesos, or its standalone manager.

• Executors: These processes execute tasks the driver assigns and report their status and results. Each Spark application has its own set of executors.

The Spark Driver-Executors cluster differs from the cluster hosting your Spark application. To run a Spark application, there must be a cluster of machines or processes (if you’re running Spark locally) that provides resources to Spark applications.

The cluster manager manages this cluster and the machines that can host driver and executor processes, called workers.

Mode

Spark has different modes of execution, which are distinguished mainly by where the driver process is located.

• Cluster Mode: The driver process is launched on a worker node alongside the executor processes in this mode. The cluster manager handles all the processes related to the Spark application.

  

• Client Mode: The driver remains on the client machine that submitted the application. This setup requires the client machine to maintain the driver process throughout the application’s execution.

  

• Local mode: This mode runs the entire Spark application on a single machine, achieving parallelism through multiple threads. It’s commonly used for learning Spark or testing applications in a simpler, local environment.

  

Anatomy

It’s crucial to understand how Spark manages the workload:

• Job: A job represents a series of transformations applied to data. It encompasses the entire workflow from start to finish.

• Stage: A stage is a job segment executed without data shuffling. A job is split into different stages when a transformation requires shuffling data across partitions.

• DAG: In Spark, RDD dependencies are used to build a Directed Acyclic Graph (DAG) of stages for a Spark job. The DAG ensures that stages are scheduled in topological order.

• Task: A task is the smallest unit of execution within Spark. Each stage is divided into multiple tasks, which execute processing in parallel across different partitions.

You might wonder about the “data shuffling” from the Stage’s part. To dive into shuffle, it’s helpful if we could understand the narrow and wide dependencies:

• Transformations with narrow dependencies are those where each partition in the child RDD has a limited number of dependencies on partitions in the parent RDD. These partitions may depend on a single parent (e.g., the map operator) or a specific subset of parent partitions known beforehand (such as with coalesce).

• Transformations with wide dependencies require data to be partitioned in a specific way, where a single partition of a parent RDD contributes to multiple partitions of the child RDD. This typically occurs with operations like groupByKey, reduceByKey, or join, which involve shuffling data. Consequently, wide dependencies result in stage boundaries in Spark's execution plan.

A typical journey of the Spark application

To celebrate Lunar New Year (the true New Year holiday in Vietnam), I’m offering 50% off the annual subscription. The offer ends soon; grab it now to get full access to nearly 200 high-quality data engineering articles.

50% off the annual subscription

Read more

To celebrate Lunar New Year (the true New Year holiday in Vietnam), I’m offering 50% off the annual subscription. The offer ends soon; grab it now to get full access to nearly 200 high-quality data engineering articles.

50% off the annual subscription

Intro

In the previous article, we examined how DoorDash, one of the largest food delivery platforms in the United States, utilizes Apache Kafka, Apache Flink, and Snowflake for their real-time processing platform. They used Flink to consume Kafka messages and write them to S3, which is later loaded into Snowflake to serve data users.

Recently, DoorDash has shared how they improved this architecture with the introduction of Iceberg. Let’s dive into DoorDash’s motivation, challenges, and benefits of this decision.

All credit for the technical details goes to the DoorDash Engineering Team. This article serves as my note after consuming their technical sharing resource.

Background

DoorDash developed an internal streaming platform to process real-time events from applications, enabling efficient support for business decisions.

At peak, the platform might receive a very high throughput workload with more than 30 million messages per second, which is approximately 5 GB of event data flowing into their system per second. These events originate from customers, dashers, merchants, or DoorDash's internal applications.

The stream platform will consume these events, process them, and write them to the associated tables in the data warehouse. Some use cases require the data to be available in near real-time.

So, how did DoorDash ensure their platform is low-latency and highly scalable?

As we recall from the previous article, DoorDash buffered incoming data with Kafka and used Flink to process and write the data to the sink.

Apache Flink is a framework and distributed processing engine for stateful computations over unbounded and bounded data streams, unlike Spark, which treats bounded data as a first-class citizen and aligns stream data into micro-batches. For Flink, everything is a stream; the batch is just a special case.

If you want to learn more about Flink, check out my article to understand its architecture and how it can achieve fault-tolerance and provide stateful processing capability.

The Flink application will consume the data from Kafka and upload it to S3 in the Parquet format. Then, DoorDash used Snowpie to copy data from S3 to Snowflake. Based on the notifications from the Amazon SQS, Snowpie will load data from S3 to Snowflake as soon as it is available using the COPY statement.

Challenges

The Flink → S3 → Snowpie → Snowflake has some challenges:

• Snowflake's cost increases when more users use the data platform. When designing this solution in the first place, DoorDash only planned for hundreds of thousands of messages at peak, which is far smaller than the current peak workload (30 million messages)

• The solution wrote the data twice, the first time is Flink writing data to S3, and the second time is Snowflake writing data to Snowflake

  

• It’s vendor lock-in (Snowflake)

To celebrate Lunar New Year (the true New Year holiday in Vietnam), I’m offering 50% off the annual subscription. The offer ends soon; grab it now to get full access to nearly 200 high-quality data engineering articles.

50% off the annual subscription

Solutions

They chose Iceberg for the new real-time data sink. DoorDash also experimented with Delta Lake, but the table format didn’t meet their expectations in terms of operational and cost aspects.

From their perspective, Iceberg can help because:

• The open table format has more mature support for Flink. In contrast, Delta Lake is more Spark-centric.

• It offers flexible schema and partition evolution.

• Iceberg has a very active community

• It supports concurrent table writes. From what I know, this feature is not exclusive to Iceberg, as all table format supports concurrent writes with optimistic concurrency control.

Architecture

With the introduction of Iceberg, the DoorDash real-time processing platform remains the same, except for the S3 → Snowpie → Snowflake pipeline. Now, the Flink continues to sink data to S3 in Parquet format, but this time these files are “wrapped” with the Iceberg metadata layer.

The pipeline that writes data to Snowflake is not necessary anymore, as Snowflake users can query Iceberg data directly on S3. This enables data consumers to continue using Snowflake to interact with the data without major changes or interruptions. In addition, DoorDash also spins up Trino clusters to query this data with the help of the AWS Glue catalog.

To implement the new solutions, Doordash needs to adjust the Flink jobs.

A typical Flink application comprises three parts: the source, the transformation, and the sink. For the new approach, DoorDash only needs to change the application’s sink to the new one that writes data to S3 in Iceberg format.

Flink provides support for an out-of-the-box Iceberg sink connector, so DoorDash only needs to make minor code changes to make things work.

Challenges when adopting Iceberg

Schema Evolution

Although the Iceberg specification supports schema evolution, the Flink-Iceberg connector does not support it and requires the table schema to be static. If the schema changes, they have to stop the Flink job, adjust the logic, and restart it.

However, with all the benefits that Iceberg could bring (more on this later), DoorDash considers this not a very big deal.

Query Performance

Some users reported that their queries were very slow compared to the original solution. In these use cases, users typically query very large nested fields with hundreds of key-value pairs. This was handled well in Snowflake with the Variant Snowflake type.

DoorDash flattens these fields in Iceberg and allocates more resources for the query workload.

Benefit

So, is it worth it?

Cost saving

With Iceberg, Doordash observed a 25-49% reduction in storage costs compared to native Snowflake storage, using only the default compression scheme (zstd).

The cost savings also come from the elimination of duplicate data writing from the original solution, which writes data first to S3 and later loads it to Snowflake's native storage.

The resources used for Snowpie are now allocated for the Iceberg operation process, such as table compaction.

The reliability and availability

The support for concurrent writes enables DoorDash to develop multiple pipelines for a single Iceberg table. This allows them to write data from different sources or have different workloads, such as a standard data sink pipeline along with the backfill pipeline at the same time.

DoorDash also enjoys the native support of Iceberg’s time travel. Although Snowflake also supports this feature, users must pay more to achieve higher data retention. With Iceberg, DoorDash can achieve time travel capabilities with more control over data retention.

The Iceberg adoption aligns with their data-lake approach, which limits the dependency on any vendor, thus providing them more flexibility. For example, they can now use other engines such as Trino to query the data.

Hidden Partitioning

Generally, partitioning a table using a transformation on a column (e.g., partition by day requires transforming the timestamp column to day and creating an extra column). Users must use this exact column to benefit from partition pruning.

For example, a table is partitioned by day, and every record must have an extra partition_day column derived from the created_timestamp column. When users query the table, they must filter on the exact partition_day column so the query engine can prune unwanted partitions. If the user isn’t aware of this and uses the created_timestamp column instead, the query engine will scan the whole table.

This is where Iceberg’s hidden partitioning shines:

• Instead of creating additional columns to partition based on transform values, Iceberg only records the transformation used on the column.

• Thus, Iceberg can save storage cost because it doesn’t need to store extra columns.

Another challenge with traditional partitioning is that it relies on the physical structure of the files being laid out into subdirectories; changing how the table was partitioned required rewriting the whole table.

Apache Iceberg solves this problem by storing all the historical partition schemes. If the table is first partitioned by scheme A and then later partitioned by schema B, Iceberg exposes this information to the query engine to create two separate execution plans to evaluate the filter again with each partition scheme.

Given a table initially partitioned by the created_timestamp field at a monthly granularity, the transformation month(created_timestamp) is recorded as the first partitioning scheme. Later, the user updates the table to be partitioned by created_timestamp at a daily granularity, with the transformation day(created_timestamp) recorded as the second partitioning scheme.

The data will be organized according to the partition scheme in place at the time of writing.

When the application queries this table using created_timestamp, the query engine applies both the first and second transformations to created_timestamp to enable partition pruning.

By leveraging Iceberg’s hidden partition, DoorDash helps users feel less confused when they need to know precisely what technical columns are used for partitioning.

My thought

One of the advice I remember the most after reading the book Fundamentals of Data Engineering is choosing common components wisely.

According to the author, data engineers should select common components that facilitate team collaboration and break down silos. They could be S3 for object storage, GitHub for version-control systems, Airflow for orchestration tools, or Spark for processing engines.

These components act like a toolkit for solving problems and prevent the need to reinvent the wheel. For the lake house specific problem, Iceberg is a strong candidate for your organization’s common component. It can work well with many systems. If you open a document from any cloud data warehouse or data processing engine, there is a very high chance that you will see Iceberg support at some level.

This provides you with more flexibility. You can make more reversible decisions. You no longer like Snowflake and want to try BigQuery, or you want to return to a self-managed, open-source solution engine like Trino. Iceberg can help you with that.

This does not mean Iceberg is the go-to choice for any data project. Every technical decision will have trade-offs, and the data practitioners should evaluate and make decisions based on the organization’s needs, not following trending tools.

Compared to using managed cloud data warehouses like BigQuery or Snowflake with their native storage offerings, adopting Iceberg requires more effort to understand how the table format works behind the scenes.

With DoorDash, I think they made a very good choice by storing data in S3 in the first place, rather than loading it directly into Snowflake. This might come from the intention of having total control over their data, but over time, this choice brings them many benefits. The most obvious one we see in this article is that it helps them onboard Iceberg more easily onto the platform.

Another observation is that we can see the advantage of “working well with many systems “ from the Iceberg, which could help DoorDash operate the Flink-Iceberg connection with just a few problems that could be easily debugged and fixed. From their sharing, DoorDash mentions more than once that they have trouble getting Flink to work with Delta Lake.

Outro

In this article, we explore the motivation for the adoption of Iceberg for their real-time process platform, including its architecture, challenges, and benefits of the new approach. Finally, I have some thoughts on the trend of adopting Iceberg.

Thank you for reading this far. See you in my next article.

Reference

[1] Tristan Culp, Gaurav Sharma, Iceberg with Flink at DoorDash (2025)

To celebrate Lunar New Year (the true New Year holiday in Vietnam), I’m offering 50% off the annual subscription. The offer ends soon; grab it now to get full access to nearly 200 high-quality data engineering articles.

50% off the annual subscription

Intro

Fourteen years ago, LinkedIn built Kafka to handle its log processing demands.

The system combines the benefits of traditional log aggregators and publish/subscribe messaging systems. Kafka is designed to offer high throughput and scalability. It provides an API similar to a messaging system and allows applications to consume real-time log events.

Now, you see Kafka everywhere. Over the years, Kafka has continued to evolve with many changes and updates. From the tiered storage to the Kraft or the queue.

But the core has remained the same since the first day. This article summarizes my learning and research on Kafka, hoping it will help you feel less overwhelmed when entering the Kafka world.

Overview

Messages

Kafka’s unit of data is called a message. A message can have an optional piece of metadata called the key. The message and the key are just an array of bytes. The key can be used if users want more control in partitioning; for example, Kafka can guarantee that messages with the same key will be placed on the same partition using consistent hashing on the key.

A message stored in Kafka doesn’t have an explicit message ID. Instead, each message is addressed by its logical offset. This avoids the overhead of maintaining index structures that map the message IDs to the actual message locations. To compute the offset of the following message, the consumer has to add the length of the current message to its offset.

Topics and Partitions

Messages in Kafka are organized into topics. A topic can be split into multiple partitions. Partitions are how Kafka offers redundancy and scalability. Each partition can be hosted on a different server, meaning a topic can be scaled horizontally across multiple servers.

Each partition of a topic corresponds to a logical log. Physically, a log is implemented as a set of segment files of approximately the same size (e.g., 1GB). Whenever a message is written to the partition, the broker appends that message to the active segment file.

Designs

Kafka use the Filesystem

Kafka lets the OS filesystem handle the storage layer. It leverages the kernel page cache mechanism to simplify the design.

Modern OS systems usually borrow unused memory (RAM) portions for page cache. This cache populates frequently used disk data, avoiding touching the disk too often. Thus, the system is much faster, mitigating the latency of disk seeks. If some application needs the memory to operate, the kernel will take back memory portions used for page cache. This ensures the page cache does not affect the system's performance.

Rather than implementing a proprietary cache mechanism, Kafka relies on the OS transferring all data to the page cache before flushing it to the disk. This approach also benefits Kafka, given the fact that it was built on the Java Virtual Machine, which has some pain points:

• The high memory overhead of stored objects.

• The garbage collector process will be slow when the number of in-heap objects increases.

Sequential access pattern

“Because the disk is always slower than RAM, is that going to affect the Kafka performance?”, you might wonder.

The key here is the access pattern. There is no doubt that with random access, the disk will be slower than RAM, but it can outperform memory slightly when it comes to sequential access. Let’s take a look at these patterns:

• Random access is a method of retrieving or storing data in which the data can be accessed in any order.

• Sequential access is a method of retrieving or storing data in which the data are accessed in a sequential order.

Kafka is designed to make writing (the producers write data) and reading (the consumers consume data) happen sequentially.

• Write: As mentioned, Kafka manages messages as segment files internally. The broker will append the message to the last segment. Appending at the end of the segment file ensures that data writing in Kafka happens sequentially.

• Read: The consumer always consumes messages from a specific partition sequentially, with the help of the two index files. The first index maps offsets to segment files and positions within the file, allowing brokers to find the message for a given offset quickly. The latter maps timestamps to message offsets; this index is used when searching for messages by timestamp.

Zero-copy

Using the filesystem also helps Kafka leverage the zero-copy optimization behind the scenes. A zero-copy operation doesn’t mean there are no data copies; it only ensures it does not make unnecessary copies. This optimization was not first invented for Kafka; it just leverages this existing technique from the OS system.

Initially, when a process reads a file from the disk and transfers it over the network, data is usually copied four times with four context switches between user and kernel modes. The flow will have the following steps:

1. It reads the file content on disk and stores it in the OS page cache. This step requires a context switch from user mode to kernel mode.

2. Data is copied from the cache into the application buffer. This requires the context to switch from kernel mode to user mode.

3. Data is then copied to the socket buffer. Once again, this requires switching the context from user to kernel mode.

4. The context is switched back to user mode after sending data to the socket buffer. It then copies the data from the socket buffer to the network interface controller (NIC).

5. The NIC sends data to the destination.

With the zero-copy optimization, the data is copied directly from the page cache to the socket buffer. In a Unix-based system, this technique is handled by a sendfile() system call. It will copy data directly from one file descriptor to another without transferring data to and from user space when using read() and write() system calls. Thus, this optimization can help Kafka bypass steps two and three from the original transfer flow:

1. The data is copied from the disk to the page cache.

2. Then, the data is copied directly from the page cache to the network controller via the sendfile() call.

3. The NIC sends data to the destination (the consumer).

As a result, the context switch is reduced from four to two, and the data doesn’t need to be copied to the Kafka application.

Batching

To make the client-broker request more efficient, the Kafka protocol has a message set abstraction that helps group messages together. This helps mitigate the network round-trip overhead when sending too many single message requests.

Batching also helps the broker write the message more efficiently; instead of appending the messages one by one, the broker appends a chunk of messages at once. This allows Kafka to achieve larger sequential disk operations.

Moreover, Kafka supports the compression of batches of messages with an efficient batching format in case the network bandwidth is the bottleneck. A batch of messages can be grouped, compressed, and sent to the broker.

This article is sponsored by Aiven. Their proposal, Apache Kafka® KIP-1150: Diskless Topics, is poised to be a game changer, aiming to reduce Kafka infrastructure costs by up to 80% through offloading disk replication to object storage. Learn more about the proposal here and leave your feedback to help shape the future of Kafka.

Producer

The flow

When you use the Kafka producer API, a few things happen:

• The process creates a ProducerRecord, including the message’s value and the destination topic. The ProducerRecord can contain a key, partition, timestamp, and headers.

• The producer will serialize the ProducerRecord’s key and value objects to byte arrays to send over the network.

• If no partition is specified, the data is routed to the partitioner; this component will choose the message’s partition based on the key.

• After knowing the destination topic and partition, the producer adds the record to the batch of messages sent to the same topic and partition.

• A different thread will send these batches to the appropriate Kafka brokers.

• When the broker receives messages, if successful, it returns a metadata object with the topic, partition, and record offset. If not, it returns an error; in this case, the producer may retry a few times before giving up.

Sending method

So, can we control the way we want to send the message? The answer is yes:

• Fire-and-forget: The producer sends a message to the server and doesn’t check if it arrives. In case of errors or timeout, messages will be lost, and the application won’t be notified.

  

• Synchronously: Sending a message synchronously allows the producer to catch exceptions if Kafka returns an error or retries fail; the producer sends the message and waits for the response. This method is rare in production because it can impact the performance.

  

• Asynchronously: The producers send all messages without waiting for replies. They support adding a callback to handle errors while executing an asynchronous send.

  

Was the message delivered successfully?

The producer exposes the acks parameter to let the user determine the successful message delivery criteria. It controls how many partition replicas must receive the record before the producer considers the writer successful:

• acks=0: The producer doesn't wait for a reply from the broker and assumes the message was sent successfully. This setting enables very high throughput. However, the risk of losing data is very high.

  

• acks=1: The producer receives a “yes“ response once the leader gets the message.

  

• acks=all: The producer gets a “yes“ response only after all replicas receive the message. This mode is the safest, ensuring the message survives even if a broker crashes. However, it increases latency.

  

How do we distribute the message?

Kafka messages can optionally have a key, which is null by default. The message’s key is mainly used to decide the message destination partition. When the key is null, and no custom partitioner is defined, Kafka will use the following:

• Round-Robin partitioner (with Kafka version ≤ v2.3): It assigns messages to partitions cyclically. It sequentially assigns messages to each partition, one after another, and then starts again from the first partition.

• Sticky Partitioner (with Kafka version ≥ 2.4): It aims to stick to a particular partition for a batch of records, meaning it tries to send as many records as possible to the same partition until a specific condition is met, such as the batch reaching its limit. Once that condition is met, it switches to another partition and continues.

If the message’s key is not null, Kafka will hash it with a hash algorithm and use the result to map the message to a particular partition. Messages with the same key will be routed to the same partition. Kafka also lets users define their custom partitioner to tailor their needs.

To celebrate Lunar New Year (the true New Year holiday in Vietnam), I’m offering 50% off the annual subscription. The offer ends soon; grab it now to get full access to nearly 200 high-quality data engineering articles.

50% off the annual subscription

Consumer

When Kafka was developed, other log-based systems, such as Scribe (from Facebook) or Flume, followed a push-based model where data is pushed to the consumers. However, LinkedIn engineers found the “pull” model more suitable for their applications because consumers can read the messages at a rate ideal for their capacity, allowing them to manage their workload effectively. The consumer can also avoid being flooded by messages pushed faster than they can manage.

The model has the following advantages:

• Catching up: If a consumer falls behind in processing messages, it can catch up at its own pace.

• Batching: Consumers can pull batches of messages when ready, enabling efficient data transfer.

The request

A consumer always consumes messages from a particular partition sequentially. If the consumer acknowledges a message offset, the broker implies that the consumer has received all the previous partition’s messages from this offset.

The Consumer API is an infinite loop for polling the broker for more data. It will issue asynchronous pull requests to the broker to retrieve the data. Each request contains the offset of the message from which the consumption begins.

The broker will use the offset to seek and return the desired data. After receiving the message, the consumer computes the offset of the following message (using the current message’s length and offset) and uses it for the subsequent pull request.

Consumer groups

Kafka has a concept of consumer groups. Each group has one or more consumers who will consume a set of subscribed topics. LinkedIn made a topic’s partition the smallest unit of parallelism; all messages from one partition are consumed only by a single consumer within a group. If the number of consumers in the group is larger than the number of partitions in a topic, some consumers will get no message.

Consumers in the same group have the same group ID. When a group ID is assigned, any new consumer instance added to the group will automatically receive this same group ID.

Kafka uses the Group Coordinator (one of the brokers) to balance the load within the group. The coordinator, determined by the group ID, ensures that messages from subscribed topics are evenly distributed among the group members. It also keeps the workload balanced when there are changes in the group membership.

When a consumer wants to join a group, they send a request to the coordinator. The first one to join the group becomes the leader. The leader gets a list of all active consumers from the coordinator and assigns a subset of partitions to each consumer. Consumers maintain membership in a consumer group and partition ownership by sending heartbeats to the group coordinator.

Partition Assignment

Each member of the consumer group will be assigned partitions to consume. Kafka has the following assignment strategies:

• Range: This is the default strategy, and it’s applied to each topic independently. It assigns a consecutive subset of partitions from each topic to each consumer. The assignor divides the number of partitions of each topic by the number of consumers to determine the assigned partitions. If it is not evenly divided, the first few consumers will have more partitions (more burden on these instances).

  

• Round Robin: This strategy works across all the subscribed topics and assigns them to the group’s members sequentially. This approach's advantage is that it maximizes the number of consumers used. If we add one more consumer to the group, each consumer will have two partitions. However, this requires a lot of partition movement in case of rebalancing.

  

• Sticky: This strategy is similar to the round-robin one used at the first assignment, but is different regarding reassignment. It tries to preserve as many existing assignments as possible when the partition reassignment occurs in the group. The strategy has two main goals: achieving a balanced assignment of partitions and minimizing the overhead during rebalancing by keeping as many assignments in place as possible.

Rebalancing

When the number of consumers changed (a member added or a member crashed), the remaining group’s consumers started consuming messages from partitions previously assigned to other consumers. The process of moving the partition’s ownership between consumers is called rebalancing. There are two types:

• Eager rebalancing: All consumers stop consuming, give up all their partition ownership, and rejoin the group to get a brand-new partition assignment. This causes a short amount of unavailability time for the entire consumer group.

• Cooperative rebalancing: This type only moves ownership of a subset of the partitions from one consumer to another and allows consumers to continue handling messages from partitions that are not reassigned.

Consumption tracking and commit offset

The unique thing about Kafka is that the consumer does not need to keep track of which message it consumes; instead, it uses the broker to track the message-consumed position. This process of updating the current position between the consumer and broker is called offset commit. The consumer will send a message to inform them that they have successfully processed messages up to a certain point. The broker will assume that the consumer processes all messages before this point. The broker updates this confirmation message to an internal topic.

The object storage trend

We learned that the Kafka design relies on the OS page cache for the storage system. This means compute and storage are tightly coupled. We can’t scale these two components independently. Scaling storage always requires adding more machines, leading to inefficient resource usage.

The design of this share-nothing architecture made sense since, in the past, networks were not as fast as they are now, and local data centers were more common than cloud resources. However, in the cloud era, Kafka’s design makes it hard to leverage the pay-as-you-go pricing. In addition, a Kafka setup could have high cross-availability-zone transfer costs due to Kafka data replication.

Although the initial designs make Kafka a very high-throughput and reliable system, it might not fit well with the cloud era. Many efforts are being made to solve these challenges. The early one is the tiered storage proposal from Uber, which allows Kafka to store messages in a two-tiered storage system:

• Local storage (broker disk) stores the most recent data.

• Remote storage (HDFS/S3/GCS) stores historical data.

However, brokers are not entirely stateless. Replication still happens, and messages still need to be moved around when the cluster’s membership changes.

Until recently, the trend of using object storage for Kafka has been emerging, from WarpStream, AutoMQ, Bufstream, to Redpanda. They made Kafka operate directly on object storage.

This approach has many benefits. Object storage is cheaper, compute and storage are separate, and data replication is eliminated because object storage ensures data availability and durability.

Recently, Aiven introduced a very powerful feature with the KIP-1150, which would forever change how we operate the open-source Kafka deployment. The KIP proposes a new class of topics in Apache Kafka that delegates replication to object storage. Users can tell Kafka to store data from a particular topic, whether on disk or in object storage.

Outro

Thank you for reading this far.

In this article, we explored Kafka’s basics, its technical designs, how the producer and consumer interact with the broker, and finally, a glimpse into the current trend of using object storage to deal with Kafka's limitations when operating on the cloud.

Now, see you in my next article.

Reference

[1] Jay Kreps, Neha Narkhede, Jun Rao, Kafka: a Distributed Messaging System for Log Processing (2011)

[2] Gwen Shapira, Todd Palino, Rajini Sivaram, Krit Petty, Kafka The Definitive Guide Real-Time Data and Stream Processing at Scale (2021)

[3] Kafka Official Documentation

[4] Wikipedia - Memory-mapped file

[5] Wikipedia - Page cache

[6] Linux ate my ram

[7] Andriy Zabolotnyy, How Kafka Is so Performant If It Writes to Disk? (2021)

[8] Stanislav Kozlovski, Zero Copy Basics (2023)

[9] Travis Jeffery, How Kafka’s Storage Internals Work (2016)

[10] Confluent Document, Kafka Consumer Design: Consumers, Consumer Groups, and Offsets

[11] Conduktor Blog, Kafka Partition Assignment Strategy (2022)

[12] Redpanda Blog, Kafka partition strategy

[13] Filip Yonov, Diskless Kafka: 80% Leaner, 100% Open (2025)

I’m making my life less dull by spending time learning and researching “how it works“ in the data engineering field.

Here is a place where I share everything I’ve learned. Not subscribe yet? Here you go:

Subscribe now

Intro

Since its open-source release, Kafka has become the de facto standard for distributed messaging. It has gone from operating only on LinkedIn to meeting growing log processing demands, now serving many companies worldwide for various use cases, including messaging, log aggregation, and stream processing.

However, it was designed at a time when local data centers were more widely adopted than cloud resources. Thus, there are challenges when operating Kafka on the cloud. Compute and storage can’t scale independently, or cross-availability-zone transfer fees due to data replication.

When searching for “Kafka alternative,” you can easily find four to five solutions that all promise to make your Kafka deployment cheaper and reduce the operational overhead. They can do this or implement that to make their offer more attractive. However, one thing you might observe over and over again is that they all try to store Kafka data completely in object storage.

This article won’t explore Kafka's internal workings or why it is so popular. Instead, we will try to understand the challenges of building Kafka on top of S3.

Background

But before we go further, let's ask a simple question: “Why do they want to offload data to S3?“

The answer is cost-efficient.

In Kafka, compute and storage are tightly coupled, which means that scaling storage requires adding more machines, often leading to inefficient resource usage.

Kafka's design also relies on replication for data durability. After storing messages, a leader must replicate data to followers. Because of the tightly coupled architecture, any change in cluster membership forces data to shift from one machine to another.

Another problem is cross-Availability Zone (AZ) transfer fees. Cloud vendors like AWS or GCP charge fees when we issue requests to different zones. Because producers can only produce messages to the partition leader, when deploying Kafka on the cloud, the producers must write to a leader in a different zone approximately two-thirds of the time (given a setup with three brokers). Kafka setup on the cloud can also incur significant cross-Availability Zone (AZ) transfer fees because the leader must replicate messages to followers in other zones.

Imagine you offload all the data to object storage like S3, you can:

• Save storage money because object storage is cheaper than disk media.

• Scale computing and storage independently.

• Avoid data replication because the object storage will ensure data durability and availability.

• Allow any broker to serve read and write

• …

The trend of building a Kafka-compatible solution on object storage is emerging. At least five vendors have introduced a solution like that since 2023. We had WarpStream and AutoMQ in 2023, Confluent Freight Clusters, Bufstream, or Redpanda Cloud Topics in 2024.

Besides all the hype, I am curious about the challenges of building such a solution that uses S3 for the storage layer. To support this research, I chose AutoMQ because it’s the only open-source version. This allows me to dive deeper into understanding the challenges and solutions.

Brief introduction of AutoMQ

AutoMQ is a 100% Kafka-compatible alternative solution. It is designed to run Kafka efficiently on the cloud by leveraging Kafka’s codebase for the protocol and rewriting the storage layer so it can effectively offload data to object storage with the introduction of the Write Ahead Log. For more details on AutoMQ, you can check my previous article.

Next, we will discuss the potential challenges of building Kafka on object storage and then see how AutoMQ overcomes them.

Latency

The first and most obvious challenge is the latency. Here are some numbers to help you imagine: with GetObject requests to object storage, the median latency is ~15ms, and P90 is ~60ms. The latency of an NVMe SSD is 20–100 μs, which is 1000x faster.

Some vendors choose to sacrifice low-latency performance. WarpStream or Bufstream believes this is a good trade-off for huge cost savings and ease of operation. These systems wait until the message persists in the object storage before sending the acknowledgment message to the producer.

AutoMQ doesn’t do that. It achieves low latency through a WAL+S3 architecture. To keep the solution low latency (write latency P99 < 10ms), the AutoMQ broker writes data to WAL. The WAL is essentially a disk device, such as AWS EBS. The brokers must ensure the message is already in the WAL before writing to S3; when the broker receives the message, it returns an “I got your message” response only when it persists in the WAL. The data is then later written to object storage asynchronously.

The idea is to use WAL to take advantage of the characteristics of different cloud storage media, which can be freely combined with S3 to adapt to various scenarios. For example:

• With EBS, WAL is optimal for low latency. However, customers are still charged for cross-AZ data transfer when producers send messages to leader partitions.

• With S3 WAL (AutoMQ treats S3 like WAL besides the primary storage), users can completely remove the cross-AZ cost, but the latency is increased in return.

IOPS

Related to the latency is the frequency of data writing to object storage. S3 Standard PUT requests are $0.005 per 1000 requests. A service with 10,000 writes per second would cost $130,000 per month.

If the brokers write the message to object storage right after they receive it from the producer, the number of PUT requests should be enormous.

To reduce the number of requests to object storage, all vendors tell the brokers to batch the data before uploading it. They buffer the data for a while or until the accumulated data reaches a specific size. Users can choose to reduce the buffer time for lower latency, but in return, they have to pay more for PUT requests.

Those brokers can batch data from different topics/partitions to help reduce the cost of writing for a single partition. In the process of batching data in AutoMQ, it may generate two types of objects:

• Stream Set Object (SSO): An object that contains consecutive data segments from different partitions

• Stream Object (SO): An object containing consecutive data segments from a single partition.

When writing the data in object storage, there are two scenarios:

• Data from the same stream can fill up the batch size and will be uploaded as SO

• Data from other partitions’ streams will be combined to meet the batch size, and the broker will upload it as the SSO.

Thus, data from a partition can be spread into multiple objects, harming the read performance when the broker issues more requests. AutoMQ has a background compaction process that asynchronously consolidates data from the same partition onto the least possible number of objects to deal with this. This ensures that data within the same partition can be stored close together physically, enabling sequential reads from object storage.

Cache Management

Following up on the latency and the IOPS challenges above, the simplest way to improve the performance of reading data in object storage is to make fewer GET requests to object storage.

Data caching can help with that; it serves two purposes: improving the data read performance and limiting the requests to object storage. But this raises another question: how can a solution manage cache efficiently to improve the cache hit? (There are only two hard things in Computer Science: cache invalidation and naming things.)

WarpStream distributes loads across agents by using a consistent hashing ring. Each agent is responsible for a subset of data within a topic. When an agent receives a request from a client, the agent identifies who is in charge of the required file and routes the request accordingly.

AutoMQ tries to keep the “data locality” characteristic like Kafka, where brokers are still aware of the partition they are in charge of. Thus, cache management in AutoMQ can be implemented by letting brokers cache data from their managed partitions. (We will discuss the data locality later)

AutoMQ manages two distinct caches for different needs: the log cache handles writes and hot reads (recent data), and the block cache handles cold reads (historical data). When brokers receive messages from producers, besides writing data to WAL, brokers also write data to the log cache to serve recent reads.

If data isn’t available in the log cache, it will be read from the block cache instead. The block cache is filled by loading data from object storage. It improves the chances of hitting memory even for historical reads using techniques like prefetching and batch reading, which helps maintain performance during cold read operations.

Metadata Management

The systems built on object storage need more metadata than Kafka. For example, Kafka can scan the file system directory tree to list Segments under a Partition. The equivalent way to do this in S3 is to issue LIST requests, but unfortunately, these requests perform poorly. In addition, because of batching data, message ordering is not straightforward like in Kafka.

These new systems need more metadata to answer questions like “which objects hold this topic’s data?” or “how can I ensure the message ordering?”

These metadata numbers correlate with the total number of objects stored in S3. To keep the number of metadata optimal, AutoMQ leverages the compaction technique from the IOPS section to combine multiple small objects into larger ones, thus limiting the amount of metadata.

In addition, Kafka uses ZooKeeper or Kraft to store cluster metadata such as broker registrations or topic configurations. WarpStream or Bufstream relies on a transactional database for this purpose.

In contrast, AutoMQ leverages the Kraft. It also has a controller quorum that determines the controller leader. The cluster metadata, which includes mapping between topic/partition and data, mapping between partitions and brokers, etc., is stored in the leader. Only the leader can modify this metadata; if a broker wants to change it, it must communicate with the leader. The metadata is replicated to every broker; any change in the metadata is propagated to every broker by the controller.

Kafka Compatibility

Besides solving all the problems above, the Kafka alternative solution must provide a critical feature: the ability to let users switch from Kafka to their solution effortlessly. In other words, the new solution must be Kafka-compatible.

The Kafka protocol is centered around an essential technical design: it relies on local disks to store data. This includes appending messages to the physical logs, dividing the topic into partitions, replicating them among brokers, load balancing, asking for leader information to produce messages, serving consumers by locating the offset in the segment files, and more.

Thus, developing a Kafka-compatible solution using object storage is extremely challenging. Setting the performance aside, writing to object storage completely differs from how they write data to disk. We can’t open an immutable object and append data to the end as we do with the filesystem.

So, how could they provide a solution using object storage to replace a solution designed to work with local disks seamlessly?

Some (e.g., WarpStream, Bufstream) decided to rewrite the Kafka protocol from scratch to adapt to object storage. They believe this approach is more straightforward than leveraging the open-source Kafka protocol.

For AutoMQ, they think the opposite. They focus solely on answering how they could rewrite only Kafka’s storage layer to reuse the open-source protocol. Although the process might encounter many challenges, I think it is rewarding. They can confidently offer 100% Kafka compatibility to the user; if Kafka releases new features, they merge them into the AutoMQ source code. But how did they develop the new storage layer to work with the object store? Let’s first revisit the Kafka internal.

In Kafka, there are essential components:

• The network is responsible for managing connections to and from the Kafka Client

• KafkaApis dispatches the request to specific modules based on the API key of the request

• ReplicaManager is responsible for message sending and receiving and partition management; Coordinator is responsible for consumer management and transactional messages; Kraft is responsible for cluster metadata.

• Storage: This module provides reliable data storage, providing the Partition abstraction to ReplicaManager, Coordinator, and Kraft. It is divided into multiple levels:

  • UnifiedLog ensures high-reliability data through ISR multi-replica replication.

  • LocalLog handles local data storage, offering an "infinite" stream storage abstraction.

  • LogSegment, the smallest storage unit in Kafka, splits LocalLog into data segments and maps them to corresponding physical files.

To ensure Kafka's 100% Compatibility, AutoMQ reuses all the logic except for the storage layer. For the new implementation, AutoMQ has to ensure that it still provides the partition abstraction so other Kafka modules like ReplicaManager, Coordinator, or Kraft can smoothly integrate.

Although Kafka exposes a continuous stream abstraction through partitions, many operations must be performed using the segment concept, such as the internal compacting process, Kafka's log recovery, transaction + timestamp indexing, or reading operations.

AutoMQ still uses segments like Kafka, but it introduces the Stream abstraction over the segments to facilitate data offloading to object storage. The stream’s core methods at the API level are appending and fetching a stream.

Compared to Kafka's Log, it lacks indexing, transaction index, timestamp index, and compaction. To align with how Kafka organizes metadata and index, AutoMQ’s stream contains:

• Meta stream provides a KV-like semantics to store metadata at the Partition level. Apache Kafka can scan the file system directory tree to list segments under a partition. In AutoMQ Kafka, Meta S3Stream uses ElasticLogMeta to record the Segment list and the mapping between Segments and Streams. This also helps avoid sending a LIST request to object storage.

• Data stream mapping between stream and segment data. It already provides the capability to query data based on logical offsets. Thus, it can replace xxx. data and xxx.index in Kafka.

• Txn/Time streams are equivalent to xxx. tnxindex and xxx. timeindex in Kafka

Unlike Kafka’s segment abstraction, which is limited to filesystem operations, a stream has more work to do, from caching messages, writing them to a write‑ahead log, to asynchronously offloading them to S3.

Convergence of Shared Nothing and Shared Disk

Both shared nothing and shared disk have advantages. The first has data locality that can serve writes and cache data more efficiently. The latter storage provides the efficiency of sharing data across different nodes. Theoretically, any broker can read and write any partitions when storing data in object storage.

With Kafka's initial shared-nothing design, partitions are bound to nodes. Read and write requests can only access the node with the assigned partitions. This is used to identify nodes to handle requests and to achieve load balancing. Thus, vendors must also consider data locality when building an alternative solution with shared disk architecture.

Warpstream, for example, bypasses the data locality for the write process; any agent in the same Availability Zone (AZ) as the client can serve the operations. When it comes to read requests, they must be served by the responsible agents. (mentioned from the Cache Management section)

Although AutoMQ is designed to store data completely in object storage, it still wants the brokers to know which partition they are responsible for. AutoMQ intends to keep the “data locality” characteristic, just like Kafka, where AutoMQ still assigns partition-specific brokers.

Throughput

A stateless broker has more things to do than a Kafka broker. In Kafka, the brokers let the OS systems handle all the storage aspects. But with a Kafka-compatible solution that runs on object storage, the broker must be responsible for buffering data in memory, uploading, compacting, or parsing data in object storage.

If not carefully designed, this can cause a lot of overhead for the broker. Compaction processes may affect regular write requests if these flows are not managed effectively.

In AutoMQ, there are the following types of network traffic:

• Message Sending Traffic: Producer -> AutoMQ -> S3

• Tail read Consumption Traffic: AutoMQ -> Consumer

• Historical consumption traffic: S3 -> AutoMQ -> Consumer

• Compaction read traffic: S3 -> AutoMQ

• Compaction upload traffic: AutoMQ -> S3

To avoid different types of traffic competing with each other under limited bandwidth, AutoMQ has classified the above traffic types as follows:

• Tier-0: Message-sending traffic

• Tier-1: Catch-up read consumption traffic

• Tier-2: Compaction read/write traffic

• Tier-3: Chasing Read Consumption Traffic

AutoMQ implements an asynchronous multi-tier rate limiter based on the priority queue and the token bucket.

Token Bucket: A token bucket is a rate-limiting algorithm that periodically refills a “bucket” with tokens, each representing permission for a request to proceed. When the bucket is empty, requests are delayed or dropped to prevent system overload.

• For Tier-0 requests, the rate limiter does not apply traffic control.

• For Tier-1 to Tier-3 requests, if the available tokens are insufficient, they are placed into a priority queue based on their priority. When tokens are added to the token bucket periodically, the callback thread is awakened to attempt to fulfill the queued requests.

Cross-AZ traffic cost

As mentioned in the Background section, the original Kafka’s design can skyrocket your cross-AZ transfer fee billing due to two main reasons:

• The producer could produce messages to the leader in different zones (1)

• The leader must replicate data to two followers in different zones (2)

With solutions built on S3, the point (2) could be resolved easily by letting the object storage take care of the data replication. For point (1), things got interesting.

Solutions like WarpStream and Bufstream tried to hack the Kafka service discovery protocol. Before the producer can send messages in Kafka, it must acquire the partition’s leader information by issuing a metadata request to a set of bootstrap servers. WarpStream or Bufstream will try to respond to metadata requests with the broker having the same availability zone as the producer, because to them, any brokers can serve message writing; there is no concept of “leader” here.

With AutoMQ, things got different because it still wants to maintain the data locality, like Kafka.

It introduced a solution where the WAL is implemented using S3 to eliminate cross-AZ data transfer costs. Imagine a scenario where the producer is in the AZ1, and the leader (B2) of Parition 2 (P2) is in the AZ2. In the AZ1, there is also a broker 1 (B1).

The producer still makes the metadata request, including producer zone information, to the set of bootstrap brokers. On the AutoMQ side, brokers are mapped across different AZs using a consistent hash algorithm. Let’s assume AutoMQ places B2 in AZ2 and B1 in AZ1. Since AutoMQ knows that the producer is in AZ1 (based on the metadata request), it will return the information of B1. If the producer is in the same AZ as B2, it will return the information of B2. The core idea is to ensure the producer always communicates with a broker in the same AZ.

After receiving the information about B1 (keep in mind that this broker isn't responsible for the desired partition), the producer will begin sending messages to B1. This broker then buffers the messages in memory and asynchronously writes them into object storage as WAL data.

After successfully writing the messages to S3, B1 makes an RPC request to B2 (the actual leader of the partition) to inform it about the temporary data, including its location (this will result in a small amount of cross-AZ traffic between brokers in different AZs). B2 will then read this temporary data back and append it to the destination partition (P2). Once B2 has completed writing the data to the partition, it responds to B1, which finally sends an acknowledgment to the producer.

Outro

Thank you for reading this far.

We start this article with the trend of building Kafka-compatible solutions on top of object storage, and my curiosity about the challenges of building a system like that. We then discuss some dimensions worth mentioning, such as latency, IOPS, and Kafka compatibility. After identifying potential challenges in each dimension, we examine how AutoMQ tries to solve them.

A quick note is that I’m not a Kafka expert at all; I’m just really interested in the system and want to share my learning with the community. So, feel free to correct me.

See you next time!

Reference

[1] Tony Solomonik, The New Age of Data-Intensive Applications (2024)

[2] AutoMQ Doc, Blog, Github Repo

[3] Warpstream Doc, Blog

[4] Bufstream Doc

I’m making my life less dull by spending time learning and researching “how it works“ in the data engineering field.

Here is a place where I share everything I’ve learned. Not subscribe yet? Here you go:

Subscribe now

Intro

When it comes to “big data”, it’s hard not to mention Google.

From MapReduce, Google File System, to BigTable.

As the business evolves, Google must continuously innovate to adapt to more data requirements. They built Spanner for transactional workload and Dremel for analytics workload. The latter is the core component of Google BigQuery, the service you might be more familiar with.

Google operates multiple services with more than a billion users worldwide. They extract insights from data produced by these services to provide better user experiences and improve quality.

Services operating users interact with this data through analytical interfaces to gain insights. To answer the user's questions, the system must process vast amounts of data and provide the results within a rigorous time constraint. Some queries must even be returned in milliseconds.

In addition to internal analytics use, Google needs to serve user-facing analytics demands with intensive requirements in terms of performance and data freshness.

This article will explore Napa, the Google warehouse system behind these analytics use cases.

Background

Before exploring Napa in detail, it would be helpful to understand the typical workloads the system is trying to serve. As mentioned, Napa doesn’t handle the same workloads you and I usually discuss when we discuss a data warehouse system.

Systems like Snowflake or Databricks usually receive batches of historical data from other systems, although these vendors support real-time ingestion. We typically expect a cloud data warehouse to help us extract insights by processing a large amount of data; we can tolerate the results provided after a few minutes and rarely need milliseconds of performance. Even if we need it, we can’t achieve it due to the characteristics of the workload: scanning a massive amount of data that needs to be aggregated and joined from many tables.

However, Napa typically serves a different type of workload. It is still analytical but less complicated (i.e, fewer joins, …), requires more high-throughput data ingestion, high data freshness, and lower response time. Data usually flows to Napa from Google services with billions of users, and the end users of Napa usually require fast analytics results to operate these services properly.

In my opinion, Napa is closer to systems like Apache Pinot or Apache Druid, which are positioned as real-time OLAP.

Requirement

Based on those typical workloads, it’s not a surprise that Napa must provide:

• Robust Query Performance: The queries must have low latency performance and low variance in latency regardless of the query and data ingestion load.

• High-throughput Data Ingestion: All Napa functions must handle heavy ingestion loads.

The third requirement is very exciting: flexibility. Google observed that Napa’s clients make a three-way tradeoff between data freshness, resource costs, and query performance. Some need data results to be highly fresh and are willing to pay more for that, while some clients can tolerate low query performance to save cost.

To see how Napa can adapt to these requirements. Let’s explore its architecture.

Architecture

The Napa conceptual design consists of three main blocks: ingestion, storage, and serving. Each was designed to handle its responsibility independently, which is essential for helping Napa adapt to clients’ flexible needs. We will revisit this point in the “Choose the trade-off sections.”

Napa leverages existing Google infrastructure components. It uses Colossus to store data, Spanner for functions that require strict transaction semantics, such as metadata management, and F1 Query for data serving. Napa has a control plan to coordinate work among the sub-services.

To optimize for its primary workload, Napa uses materialized views as the main technique to maximize query performance.

This is different from systems like Snowflake or Databricks, which rely on the ability to efficiently prune unnecessary data (e.g., using a min-max index like the one in Parquet). Napa’s materialized views are sorted, indexed, and range-partitioned by primary key(s).

Napa implements LSM-trees for its storage engine to achieve high data ingestion throughput.

We will discuss these technical designs in more detail in the following sections.

Ingestion

The goal of the ingestion component is straightforward: insert large volumes of data into Napa’s storage. It accepts data, performs some lightweight transformation, and writes data (e.g., successfully writes to disk or replicates to other data centers).

Napa allows users to increase or decrease the number of data ingestion workers.

Storage

This block's main responsibility is to store the data, and because materialized views are used to boost query performance, it is also in charge of view maintenance.

Google employed two important technical designs for storage. First, Napa used two file formats: The write-optimized (WO) format serves high-throughput data writing, and the read-optimized (RO) format is designed for efficient data reading.

Second, Napa uses the LSM-tree (Log-Structured Merge-Tree) paradigm to handle table and view maintenance.

LSM-trees employ an append-only strategy optimized for write throughput; writes are buffered in an in-memory component (memtable) and periodically flushed to disk as immutable, sorted segments (SSTables). It relies on a background compaction process to merge these segments, reconcile updates/deletes, and maintain sorted order across levels, thus converting random writes into sequential disk I/O at the cost of potentially higher read amplification (checking multiple segments) and write amplification (during compaction).

The other paradigm you might be more familiar with is B-trees. These are mutable, page-oriented structures that perform updates in place, meaning writes (inserts, updates, deletes) navigate the tree to find the specific disk page containing the relevant key range and modify it directly. This often incurs random I/O, potentially triggering page splits or merges to maintain balance.

I will not deep dive into LSM-trees or B-trees here because those could require dedicated articles. So, see you in my future articles for these topics.

Back in Napa, the data is ingested into the WO format. These files might not be immediately available for reading because they can impact the query performance. Later, the data in WO format will be converted into RO format. The RO format shares some characteristics with the PAX file format, which can be found in Parquet, BigQuery, or Snowflake’s file format. The WO-RO conversion or merging multiple small RO files process is implemented via the LSM trees compaction process.

The LSM tree relies heavily on the compaction process, which improves query performance and reduces storage consumption by

• Sorting records based on key(s) to allow binary search.

• Aggregating multiple updates to the duplicate rows to avoid jumping around to find all the updates. Because in LSM-tree, deletes and updates are treated as inserts, there is a high chance that updates for a single record are scattered across multiple files. This differs from B-tree, where data is updated in-place, so fragmentation is not a concern like in LSM-tree.

Because data is sorted in each segment, compaction is essentially merge sorting. Napa applies the compaction process for both table and materialized view updates. A note here is that the process of aggregating updates for records can also happen at the serving times if the LSM compaction process can not handle it before the query engine processes this data for the query.

Serving

For many Napa clients, obtaining query results within milliseconds is critical for their business use cases. Google has employed many techniques for Napa to achieve this:

• As mentioned, Napa aggressively leverages materialized views. Napa uses views to answer a query whenever possible instead of the base tables.

  

• Filters are pushed down to the storage layer to minimize the data transferred via the network.

  

• Napa also relies on parallelism to reduce the data each subquery has to read. Each segment in the LSM-tree keeps a local index. Napa uses this index to partition an input query into thousands of subqueries that satisfy the filters.

  

• Napa maintains two cache layers to limit disk accesses: the first is the local RAM of the workers who process the query, and the second is the distributed caching layer, which can be shared between workers.

  

• Napa also prefetches data to reduce the number of disk accesses further.

  

• The systems combine small I/Os as much as possible to improve the efficiency.

  

Choose the trade-off

As mentioned, Napa allows users to trade off one of performance, freshness, or cost to achieve the remaining two dimensions. Let's first understand how Napa could achieve each dimension:

• High Freshness: As mentioned above, data written in WO format is not available immediately. If users need fresher data, Napa must speed up the conversion/compacting of the newly written data to serve it faster. There are cases when the query requires files that should be merged during the background compaction process; however, for some reason (e.g., fewer resources for the compaction), these files are still not merged at the query time, and the merge must be executed at runtime instead to ensure the freshness requirement.

• Higher Performance: Napa maintains more views to speed up query performance. More views mean more workers are needed to update them. Another requirement is to optimize the data's physical layout; too many small files can harm performance. Also, the query performance would be degraded because the file merge process could be executed at runtime instead.

• Low Cost: The system's resources can be roughly categorized into these workloads: compaction process, View maintenance, and Data Ingestion. The less the resource is used, the lower the cost.

Giving the decouple architecture of Napa, tuning to optimize for two dimensions, and sacrificing the remaining is straightforward:

• If the client wants to sacrifice data freshness for moderate query performance and cost, Napa can maintain a moderate number of views and fewer files to merge at query execution time. It uses fewer workers and cheaper resources (like spot VM instances from AWS or GCP) for view maintenance to keep costs low. Thus, the view maintenance occurs more slowly; hence, the data is not so fresh. However, clients still get pretty good query performance and keep the resource cost down.

  

• If the client wants to sacrifice query performance for high data freshness and low cost, Napa can maintain fewer materialized views but allow more files to be merged during the query runtime. Napa coordinates more workers for the ingestion process because the view maintenance effort is low. That said, clients can get fresher data results with the trade-off for the query performance because the query engine has fewer materialized views for reference, plus it has to merge more files at execution time.

  

• If the client can tolerate high resource costs for both good query performance and high data freshness, Napa can direct more workers to ingestion, data compaction, and the view maintenance process. The data will be ingested with higher throughput, data in the LSM-tree will be merged faster, and the materialized view will get updates more frequently, thus the clients will have the desired query performance and data freshness. In return, the resource cost will be relatively high.

  

Outro

Thank you for reading this far.

In this article, we learn the motivation behind Napa and the requirements it must adapt to. We then explore the system's architecture and the technical designs that Google made to make Napa a highly scalable, high-throughput, robust, and extremely flexible system.

It’s not hard to point out the shared characteristics between Napa and other analytics systems:

• Vortex, the BigQuery storage engine, also has two file formats to serve data write and read separately. The system also uses LSM-tree to manage table data like Napa did.

• Clickhouse uses an LSM-tree-like mechanism to achieve high throughput data for the MergeTreeEngine.

• Apache Hudi also has two file formats.

• The new player table format, Apache Paimon, adopts an LSM tree for the file storage.

• RisingWave with its LSM storage engine, Hummock

• …

All these systems are designed for high-throughput data writing and low-latency query performance. The case of Vortex is more special because Google decided to build Vortex for BigQuery after years of operation, when they wanted to offer real-time analytics for users, and concluded that the legacy batch storage engine couldn’t provide what they needed.

I think we will see more systems like this in the near future, given that real-time analytics data is getting more and more attention.

What do you think about this observation?

—

Now, see you in my next articles!

Reference

[1] Google, Napa: Powering Scalable Data Warehousing with Robust Query Performance at Google (2021)

[2] Jagan Sankaranarayanan, Google Napa: Scalable Data Warehousing with Robust Query Performance (2021)

I’m making my life less dull by spending time learning and researching “how it works“ in the data engineering field.

Here is a place where I share everything I’ve learned. Not subscribe yet? Here you go:

Subscribe now

Intro

We’re living in a time when it’s getting easier for data practitioners to build data pipelines. Cloud data warehouses are getting more and more powerful. The introduction of dbt streamlines data transformation using SQL.

However, that does not mean the above pattern has no challenges. We must set up the traditional orchestrator environments and determine how to schedule dbt tasks. If you use the free version of dbt, you must write a custom operator by yourself, as there is only support for the dbt cloud operator.

These tasks are not easy.

Realizing these hassles, Orchestra, a complete Data and AI workflow solution, offers us a more efficient way to operate the end-to-end data pipeline.

Motivation

The idea of Orchestra is simple:

Giving everyone the power to build and manage Data and AI workflows, even if they have little engineering experience.

Orchestra aims to democratize the ability to build, deploy, and monitor pipelines, which was the main responsibility of highly skilled data engineers in the past.

With Orchestra, we only need to log in to the platform and set up how to connect with external systems like dbt or the cloud data warehouse, and then we can start building the first data pipeline.

It’s an efficient, declarative framework for defining DAGs, with the option to use Python/dbt. Orchestra abstracts away all the complexity for users and exposes only modern UI/UX for all operations.

Let’s build a data pipeline

This section will walk you through the step-by-step process of building a data pipeline on Orchestra. We will also explore Orchestra concepts and features.

A brief on the data pipeline

You can find all the related code in this repo.

In this project, we will need an Orchestra account, a Snowflake (trial) account, and an S3 bucket to follow along.

We will use a Python script to load CSV files to S3.

We use five AdventureWorks sample datasets: product, product_category, product_subcategory, sale, and territories. You can find these files in the repo.

Then, we load these tables into Snowflake and set up a dbt-snowflake project for the transformation.

All the tasks will be scheduled using Orchestra. You can check the Python script here and the dbt-snowflake project here.

In the scope of this article, I won’t dive deep into how you could set up your Snowflake warehouse or the dbt project. If you want to learn, here are some good resources to get started:

• Snowflake quick start

• dbt core quick start

• dbt-snowflake set up

Set up the integrations

Like a traditional orchestrator, such as Airflow, one of the first things you want to do before building a data pipeline is to set up how you connect with the external systems. You would add some connections and maybe write some custom hooks, which can be time-consuming.

Orchestra provides managed integrations that take care of auth, error handling, triggering, polling, and metadata gathering out of the box. We set up “integrations,” which are connections to external systems. They support a wide range of integrations:

• ETL tools: Airbyte, Fivetran

• Data Warehouse: Databricks, Snowflake, or BigQuery

• Cloud Services: common tools in AWS, Azure, and GCP

• BI tools: Power BI, Tableau, Sigma, Lightdash, etc.

• Transformation: dbt core, dbt cloud, or Coalesce

• Utility functions: Python, http, etc.

To create a new integration, we click “Integrations“ from the sidebar and then choose the needed integration from the UI.

For this project, we need to define three integrations: Snowflake, dbt, and Python.

• The Python integration requires the connection name, the repo containing the Python scripts, and the token to help Orchestra have read access to the repo. You can check this guide to get the GitHub token.

• The dbt-core integration requires the connection name, the repo containing the dbt project, the GitHub token, and the dbt profile.

• The Snowflake integration requires the connection name and the Snowflake warehouse information.

Set up the pipeline

Like Airflow, there is a concept called “task,“ the most basic execution unit in Orchestra. Tasks leverage integrations to interact with the external system and execute the user-defined logic.

Users arrange tasks into Pipelines, specifying the upstream and downstream dependencies. In other words, Pipeline lets Orchestra know in which order it should run your tasks.

To build the pipeline, we choose Pipelines in the sidebar → New pipeline → + Create new Pipeline:

• After this, you will see two options: Orchestra and GitHub. At this time, we will go with the Orchestra. This option lets you build the data pipeline directly on Orchestra’s UI.

  

• In the next step, Orchestra prompts you to choose the type of trigger you want. Available options are manual, triggering by webhook, triggering by other pipelines, triggering by sensor, or cron jobs. For the sake of simplicity, we will go with the manual option. We can change or add trigger types later.

• Next, we will add the first task by clicking “Add task” in the Task Group. Tasks in the same Task group will be run in parallel.

  

• The first task is the Python task, which uses the defined Python integration. Orchestra will ask us to “Choose an integration job.” For this task, we will go with the “Python-execute script.”

  

• The task will run this Python script to upload data from the local to the S3 bucket. This task also needs some environment variables to work with the S3 client (e.g., AWS_ACCESS_KEY_ID, etc). We enter some information for this task, and the values of the environment variables will be included in the section “Environment Variables JSON.“ Orchestra will encode these variables when we save them.

• Next, we set up a Snowflake task that runs queries to load the CSV from the S3 bucket. We click “Add task“ in the next task group to set the dependencies between this task and the Python task. Before this task, we had to set up a few things from Snowflake so it could read the data in S3.

• We choose Run Query (Snowflake)

  

• Enter the task’s name, the SQL needed to be run, and the defined Snowflake connection. You can check the SQL script here.

  

• The two final tasks will be dbt tasks. The first one will run all staging models to clean the data loaded from the S3 bucket. The latter will run all curated models to transform data from staging into fact and dimension tables. For the dbt staging task, we choose the dbt Core command task.

  

• Then, we enter some required information for this task:

• For the dbt curated task, we will configure it the same as the dbt staging task, except for the dbt commands, which need to be changed to dbt build -s tag:curated

And that’s it; we built a pipeline with Orchestra. To recap what we’ve done, you can check the video here:

Run the pipeline

After having the pipeline’s tasks can be run in one of the following ways:

• Run based on the trigger configuration.

• Run the whole pipeline manually.

  

• Run a specific task.

  

Observability

A very cool feature of Orchestra is that the platform will aggregate all the metadata for us.

After running the pipeline, Orchestra will display the status of that run for us; in the screenshot below, we can check the status of each task. If a task has any issues, we can detect them from here, fix them, and re-run the pipeline.

Upon the pipeline finishes, we can click the “Explore lineage“ button to explore its lineage.

Orchestra also keeps the history of every pipeline’s run.

In addition, Orchestra also collects metadata about our data assets. In our project, we made five Snowflake tables and eight dbt models.

For the native Snowflake tables, Orchestra can automatically collect the metadata; for the dbt models, we must explicitly allow Orchestra to do that. We can go back to the pipeline and edit the dbt tasks to let them collect the metadata:

We can check all the assets from the sidebar in the Data Assets section. For my pipeline, the data assets include metadata like Snowflake’s table structure (from the left), the number of assets, the asset coverage, the asset health, or the asset listing.

Environment

When we build the pipeline, we want different development environments (e.g., dev, staging, and prod). Airflow requires us to set up multiple instances and multiple compute instances (e.g., two Spark clusters for the dev and prod environments).

Orchestra provides a simpler way; an environment is just a configuration. When defining a pipeline, users can add this configuration in each task to specify the different environments it can run.

For our project, imagine we need to separate the environment: “develop” and “production” for the Snowflake and dbt tasks. We want the same pipeline to run on two different Snowflake warehouses. To achieve this in Orchestra, we need:

• Two Snowflake and two dbt integrations are associated with the “develop” and “production” environments.

• Defining “develop” and “production” configuration environments in Orchestra UI, including associated integrations for each environment

• Use these configurations in desired tasks.

After this, whenever you run the pipeline or a single task, it will run in a specific environment—the default one or the one you choose. Orchestra will spin up separate resources and align these across environments behind the scene for us.

With this approach, we can control which tasks should be run in different environments and which are acceptable to run in one environment.

Orchestra also allows us to trigger the pipeline manually via GitHub workflow, which helps us integrate deeply with the CI/CD pipeline; let's say the current pipeline has two environments: “develop” and “production”:

• I want whenever I push dbt’s changes to the “develop” branch, it will run the Orchestra pipeline in the “develop” environment to validate the change.

• If there are no issues, I will merge changes to the “main“ branch and run the Orchestra pipeline in the “production” environment.

Orchestra provides the orchestra-hq/run-pipeline GitHub Action to let users integrate into the GitHub workflow. All we need is the Orchestra API key and the pipeline’s ID (from the URL).

For this project, I prepared a GitHub workflow as follows:

The workflow has two jobs. The first will check if there is a pull request from the develop branch; if so, it will run my pipeline in the “develop“ environment. The latter will check if there is a push into the “main“ branch; if so, it will execute the pipeline in the “production“ environment.

A very cool thing is that the output, when running from the pipeline, will be streamed to the UI when we observe the running workflow from GitHub.

Version Control

The level of how Orchestrate can integrate with Git does not stop there. When we create the pipeline, Orchestra will record the pipeline definition in a YAML file. Whenever we make changes to the pipeline in the UI, it will show changes just like Git:

Orchestra gives us the ability to version control this YAML file with Git. To do this for our project, we need to:

• Allow Orchestra to have access to your Repo. We go to Settings → User Git-control settings → Connect. It will ask if you allow Orchestra to have some specific permissions.

• Edit the pipeline, click the Cogwheel symbol, and choose GitHub for pipeline storage. We then fill in some Git’s information, click update, and click save.

• There will be a pull request to add the Orchestra pipeline’s YAML file to your repo.

• From now on, we can change the YAML file in the repo, and those changes will be reflected in the Orchestra pipeline UI.

In the video below, I configured the version control for the Pipeline YAML file in my repo and adjusted the file from my IDE to add another DBT task. Let's see how it works:

Access Control

To implement access control, we created a Group with associated permissions. Orchestra lets us define permissions on high-level resources like account settings to low-level ones like pipelines or environments.

After that, we will add users to suitable groups. When in a group, users will be granted all the permissions associated with that group

My thoughts

Orchestra is built for scale and makes building modern data workflows super easy. It provides a much more seamless way to build data pipelines, especially the “modern“ ones with dbt + cloud data warehouse; everything can be done on its intuitive UI/UX. The ability to collect, aggregate, and display metadata is also really valuable.

Regarding operating in different environments, Orchestra also does a very good job of abstracting the complexity of infrastructure management behind the scenes and only letting users operate on some configurations. The GitHub integration also impresses me a lot.

These are only some of my first impressions of Orchestra. There is a lot of flexibility and features I did not explore fully, like how to support multiple domains and effectively govern access to different resources in large enterprise organizations - these are very difficult using traditional workflow orchestration platforms.

Of course, all the tools will have pros and cons, and I believe that if I spend more time with Orchestra, I can spot some points that need to improve.

With this platform, it is sure that you can’t have the flexibility you got with Airflow, where you can write custom operators and construct the DAG in Python; in return, Orchestra will abstract all the complexity away, while you can build a robust pipeline from the UI or in code.

I think Orchestra is really worth your time trying, especially if you have limited resources on your team or just want to spend time on business logic instead of maintaining an workflow orchestration system.

Outro

Thank you for reading this far.

In the last 10 minutes, we have explored Orchestra and its motivation. We have also built a pipeline and tried out some very cool features from the platform. Finally, these are my naive thoughts on Orchestra.

Now, it’s time to say goodbye. See you in my next article.

Reference

[1] Orchestra Documentation

[2] Orchestra website

[3] Product-Demos

My ultimate goal is to help you break into the data engineering field and become a more impactful data engineer. To take this a step further and dedicate even more time to creating in-depth, practical content, I’m excited to introduce a paid membership option.

This will allow me to produce even higher-quality articles, diving deeper into the topics that matter most for your growth and making this whole endeavor more sustainable.

Upgrade subscription

Intro

This week, we will explore how Uber engineers build ETL pipelines to support the internet-scale business.

Uber is the tech company that transformed the taxi market in the early 2010s. In 2023, 137 million people used Uber or Uber Eats once a month, and Uber drivers completed 9.44 billion trips.

This article will first discover the company business's requirements for data pipelines and how Uber delivered the solution.

Business Requirement

At Uber, data is unified in a centralized petabyte-scale data lake. The Global data warehouse team is in charge of building foundation fact and dimension tables on this lake, acting as Lego pieces for all the data use cases, from reporting to machine learning.

The data is not only used for common analytic cases; Uber also uses data to power critical functions of their services and applications, such as rider safety, ETA predictions, or fraud detection.

For Uber, data freshness is the backbone of the business; they invested heavily in the ability to process data as soon as it’s captured to reflect changes in the real world.

They build, evolve, and manage their data lakehouse to ensure it can do one thing efficiently: handling data incrementally.

Let’s review a typical use case at Uber to understand why incremental data processing is essential.

The use case is the driver and courier earnings. Imagine Uber had a dataset containing daily driver earnings for every driver. A rider can tip the driver hours after a trip is completed. A driver earned $10 for the trip on Monday night and got an extra 1$ tip on Tuesday morning.

With batch processing, Uber doesn’t know if the driver’s earning data will be changed. They have to assume that “Data was changed in the last X days“ and reprocess all X data partitions to update the driver earnings. A small change can cost them a lot of time and resources to re-process the whole month of data (for example)

With another use case where merchants can update the menu whenever needed, Uber has to ensure these changes are reflected on the Uber Eats app. For a given day, Uber observed 408 million delta changes compared to 11 billion total entities, roughly 3.7%.

The batch approach could result in the same problem as the use case above: a small fraction of updates can waste time and resources running the pipeline over a large amount of data, leading to data freshness SLA violation.

What if they could extract only the update (e.g., the event where the rider tipped $1$) and apply it to the target table?

Apache Hudi

To bring the incremental processing capability to the lakehouse, Uber developed the Apache Hudi table format. In the scope of this article, I won’t dive deep into the story behind Hudi. If you want to read more about its story and features, check out my previous article:

In short, Hudi has a very special design compared to the Iceberg or Delta Lake format. The ultimate goal of it is what you see over and over again in this article: processing data incrementally as efficiently as possible. To achieve this, there are Hudi’s characteristics that we need to be aware of:

• Two file formats: The base files store the table’s records. To optimize data reading, Hudi uses a columnar-oriented file format (e.g., Parquet) for the Base Files. The log files capture changes to records on top of their associated Base File. Hudi uses a row-oriented file format (e.g., Avro) for Log Files to optimize data writing.

  

• Timeline: Hudi Timeline records all actions performed on the table at different times, which helps provide instantaneous views of the table while also efficiently supporting data retrieval in the order of arrival.

  

• Primary key: Each record in a Hudi table has a unique identifier called a primary key, consisting of a pair of record keys and the partition's location to which the record belongs. Using primary keys, Hudi ensures no duplicate records across partitions and enables fast updates and deletes on records. Hudi maintains an index to allow quick record lookups.

  

Hudi at Uber

This section will explore in detail how Uber implements Hudi for their Lakehouse.

Data Read

Hudi supports these types of queries:

• Snapshot: The queries will see the latest snapshot of the table.

  

• Time Travel: The queries will read a snapshot of the past.

  

• Read Optimized: This one is similar to the snapshot query but performs better because Hudi will read the snapshot using only the columnar files.

• Incremental (Latest State): The queries only return new data written since an instant on the timeline.

  

  A Hudi instant is a point-in-time marker in Apache Hudi’s timeline that captures a single atomic action (such as a data commit or compaction)

• Incremental (CDC): This is a variant of the Incremental one where it provides database-like change data capture log streams out of Hudi tables

  

Uber uses Incremental (Latest State) most of the time to handle many types of reads and joins with Hudi:

• Incremental update from a single table: Uber reads data incrementally from the Hudi source table and uses this data to update the target table.

• Consolidation from single table incremental update and other raw tables: To prepare for the updated data for the target table, Uber reads data incrementally from the Hudi source table and performs left outer join on other raw data tables with T-24 hr incremental pull data.

• Consolidation from single table incremental update and other derived and lookup tables: Uber reads data incrementally from the Hudi source table and performs left outer join on other derived tables with only the affected partitions

• Backfilling: Uber leverages Hudi’s snapshot read on single or multiple source tables using the backfill start and end date boundaries.

Data Write

In Hudi, write operations can be classified into two types:

• Incremental: Hudi applies only incremental changes to the table/partition.

• Batch: Hudi overwrites entire tables and/or partitions entirely every few hours.

For each type, Hudi further categorizes operations into these types:

• Upsert (Incremental): Hudi first looks up the index to check whether the record is tagged as inserts (new) or updates (existing). Then, Hudi determines how to pack the record in the storage. The target table will never show duplicates.

  

• Insert (Incremental): This one resembles Upsert, but Hudi skips the index-look-up step. This option is faster than Upsert; however, the target table can show duplicates.

  

• Delete (Incremental): Hudi supports two types of deletes on Hudi table data. Based on the record key, Hudi can soft delete where it retains only the record key and fills null for all the other fields. The other approach is hard delete, which entirely clears all evidence of a record

  

• Bulk Insert (Batch): Insert or Upsert keeps data in the memory to speed up computations, which can cause some problems for initial data loading. Bulk insert has the same semantics as insert plus implementing a sort-based data writing algorithm, which can scale well for initial data load.

  

• Insert Overwrite (Batch): Hudi will rewrite all the partitions that are present in the input.

  

• Insert Overwrite Table (Batch): Hudi will rewrite the whole table.

  

To write data to Hudi tables, Uber has to handle it differently based on whether the table is partitioned or not:

Hudi stores data files under partition paths for partitioned tables (like Hive table) or under the base path for non-partitioned tables. For example, Hudi organizes table_1, partitioned by date, in folders like table_1/date=2025-04-01, table_1/date=2025-04-02,….For non-partitioned tables, Hudi stores it using only the base path: table_2/.

• Partitioned tables: Uber uses upserts to apply the incremental updates. For backfilling, they use insert_overwrite to rewrite the affected partition. For non-incremental columns, they use Spark SQL targeted merge/update statements.

  A non-incremental column is any column whose updates do not determine how a record changes over time in the sense of incremental data loads (e.g., a restaurant located in Las Vegas last year and later changed to New York).

• Non-partitioned tables: Uber also uses upserts to apply the incremental updates. To update the incremental and non-incremental columns, they use insert_overwrite when joining (full outer join) incremental rows with the target table.

The actual implementation

Uber handles the incremental ETL pipeline using Hudi, Spark, and its internal data workflow, Piper (think Airflow). They built a Spark ETL framework to manage ETL pipelines at scale, using Hudi’s incremental data processing tool called DeltaStreamer to power this framework.

Uber initially contributed to DeltaStreamer, and many organizations have used it to streamline incremental data processing with Hudi. In more detail, the tool provides ways to ingest from different sources, such as Kafka.

The Spark ETL framework abstracts all the complexity and lets users configure how their pipeline should run with simple steps. Users must give the framework a few inputs, like the table definition, DeltaStreamer YAML configs, and the SQL or Java/Scale transformation.

• Table definition: A DDL definition file with the table’s schema information and Apache Hudi format.

• DeltaStreamer YAML configs: This file will provide a list of configurations expected by the Apache Spark DeltaStreamer application. Some important ones are the hoodie.datasource.recordkey.field, which declares the target table’s primary key. As mentioned, Hudi uses the primary key to perform data duplication (with the upsert write operation). The next important one is hoodie.datasource.write.operation, which expects one of the values listed in the “Data Write“ section above.

• Transformation logic: The user will provide a file with the SQL transformation logic. The DeltaStreamer will execute this logic using Spark SQL. Users must specify the incremental source from which the DeltaStreamer performs the incremental read operation. The tool will read from the latest checkpoint in the target table’s Hudi metadata to capture the new data. Users can express the transformation logic using Spark Scala/Java for more advanced use cases.

Impact

Performance and Cost Savings

Because of migrating all the batch ETL pipelines to the incremental solution with Hudi, Uber decreased the pipeline run time by 50%. I captured the table from Uber’s article to show how efficient the new solution is:

They used 59.06% CPU core and 73.01% memory less than the ETL approach for the Dimensional Driver Table. In the past, the pipeline would take roughly 3.7 hours to finish; with the incremental pipeline, it only takes Uber 39 minutes to finish.

Data Consistency

To achieve availability, Uber organized data redundantly across multiple data centers. Achieving strong data consistency across tables in different data centers is critical to Uber’s business operations.

Hudi helps Uber consistently replicate data across data lakes in many data centers. After computing the table in the primary center, Uber replicates the data by using the Hudi metadata to move incrementally changed files across data centers.

Data Quality

Uber implement the write-audit-publish (WAP) pattern with Hudi to prevent low-quality data from entering the production environment. This approach requires users to run SQL-based data quality checks on the data before it gets pushed to the production dataset.

Observability

The Hudi’s DeltaStreamer outputs valuable metrics to provide insights during the incremental ETL processes. Uber can observe the number of Hudi’s commits in progress or the total records inserted/updated/deleted.

Outro

Thank you for reading this far.

In this article, we explored why incremental processing is critical to Uber’s business and how Uber solves the problems with Apache Hudi.

For me, Hudi is an exciting table format with many interesting technical designs. Although it does not get wide adoption like Iceberg or Delta Lake, Hudi will shine in the use cases it was originally designed for.

Would you like to read more Hudi articles? If yes, please let me know in the comment section or leave a reaction to this article.

Now, it’s time to say goodbye.

See you in my following articles.

Reference

[1] Uber Engineering Blog, Setting Uber’s Transactional Data Lake in Motion with Incremental ETL Using Apache Hudi (2023)

[2] Hudi Write Operations

[3] Table & Query Types

To celebrate Lunar New Year (the true New Year holiday in Vietnam), I’m offering 50% off the annual subscription. The offer ends soon; grab it now to get full access to nearly 200 high-quality data engineering articles.

50% off the annual subscription

Intro

In this article, we will explore how Meta, one of the world's biggest tech companies, re-architected its data lakehouse. The texts you’ll read will not cover detailed components of the Meta lakehouse. Instead, we will see how Meta’s initial approach caused them troubles and their effort to fix them at the organizational scale.

For this article, I referred to material from the Meta paper released in 2023 called Shared Foundations: Modernizing Meta’s Data Lakehouse.

The initial approach and its problems

Meta started their data journey about +20 years ago.

They started implementing the paradigm of bringing the query engines to the data stored in object storage with Hive in 2010. It was eleven years before Databricks released the paper introducing the lakehouse architecture.

Since then, Meta's warehouse system has grown from tens to hundreds of petabytes, and in 2023, it reached multiple exabytes. With Hive, the Meta high-level warehouse solution can be described as below:

• They managed data, metadata, and computing independently.

• They stored data in HDFS, which let them scale the storage layer independently from the computing layer. In recent years, Meta has replaced HDFS with an in-house file system called Tectonic, which helped them achieve operational efficiency.

• They stored metadata in the MySQL database. With Hive, users can store partition information in the Hive Metastore.

• They store data in a columnar format. They first created the RC file and later enhanced it to create ORC. They also developed an ORC variant called DWRF to support nested data and encryption better.

• Internal users can bring their favorite compute engine to join the party. From Spark, Presto to Meta deployment of Giraph - an iterative graph processing system.

However, this architecture caused Meta some problems:

• This architecture did not support stream processing. This made Meta build various streaming systems over time, and of course, they were not so well integrated with Hive.

  

• The architecture did not support real-time data ingestion to Hive. They ended up using Scuba for this purpose, although it was initially built for log analytics.

  

• There were a lot of programming languages. Most of the data stack in Meta was written in Java, but most of the other systems in Meta use C++. Java is also not primarily supported at Meta.

  

• Hive was too slow for interactive queries. Meta had to create new engines to address this problem. They wrote some in Java and some in C++. Even though some engines were written in the same language, they did not share any components, resulting in solution fragmentation.

• At first, Meta stored data in HDFS storage nodes, mostly using HDD for local disks. For interactive queries, fetching data from HDD over the network is slow. Meta had to develop many interactive query engines that had compute and storage tightly coupled to improve query latency. This caused the solution fragmentation and data deuplication to become more serious.

• The fragmentation did not stop there. At Meta, there were at least six SQL dialects, three implementations of Metastore client and ORC codecs, about twelve different engines targeting similar workloads, and many copies of the same data in various locations and formats.

  

Meta lacked the standardization and reusable components. The engineers got more operational burden. The users had to interact with different SQL dialects and suffer inconsistent semantics.

They couldn’t put the most effort into innovation.

The new paradigm shift

So, how did Meta solve those problems?

They started an effort on an organizational scale, which Meta called the Shared Foundations. The purpose is to re-architect the data lakehouse.

The Shared Foundations program involves hundreds of engineers throughout Meta. The program has the following principles:

• Using fewer systems: Many systems that serve the same use cases with overlapping functionality should be merged into one system. For example, Meta aimed to have a single query engine for each area: batch, streaming, interactive, and machine learning.

• Reusable components: Meta can still provide different compute engines if use cases and requirements are distinct. They focused on reusing as many components as possible for these cases. For example, interactive and batch engines can share the storage encodings or data formats.

• Consistent APIs can lower the learning curve for users and make the integration of components more straightforward. Thus paving the way for modularization and reusability.

With these principles, Meta aimed to achieve:

• Engineering efficiency: Their engineers can work on a smaller number of systems. These principles also reduced duplication and prevented them from re-inventing the wheel.

• Faster innovation: Having fewer systems means less operational burden. This allows Meta to focus on new features and other improvements.

• Better user experience: End users can expect consistent syntax, features, and semantics across systems, lowering the barrier to using these systems and increasing productivity.

Meta implemented the Shared Foundations in areas such as storage, metadata, execution, language, and engine.

To celebrate Lunar New Year (the true New Year holiday in Vietnam), I’m offering 50% off the annual subscription. The offer ends soon; grab it now to get full access to nearly 200 high-quality data engineering articles.

50% off the annual subscription

Compute Engine

As mentioned, internal teams built different query engines to adapt to different workloads and performance requirements.

Presto, Raptor, Cubrick and Scuba for interactive queries.

Presto and Spark for batch execution.

Puma, Stylus, XStream, and MRT for stream processing.

Let’s dive into each area.

For the interactive engines, the ideal one would have the best features from Presto, Raptor, Cubrick, and Scuba. This engine should provide:

• Full SQL support, complex queries, and data models.

• The ability to process data directly on the lakehouse.

• Low latency performance is achieved by managing data in memory or SSD.

• Supporting for real-time data.

In the end, Meta built the convergence engine based on Presto because it provides most of the requirements above. Meta compensated the performance gap between Presto and other interactive systems through local caching.

They developed the smart hierarchical caching mechanism, which stored the most frequently used data and metadata in the local memory and SSDs of Presto’s workers and coordinator.

This mechanism helps improve the order of magnitude of the latency of most of Meta’s common interactive query patterns. This speedup even exceeded the performance of existing systems, which use less hardware.

Although Presto can query directly near real-time data on Hive, the engine can only tap into real-time data partitions once all the partition’s data is available. A Hive partition typically has hourly or daily partitions, limiting the near-real-time capability.

To address this, Meta introduced the open partition state in the Hive Metastore as the systems could register the partitions as soon as the data arrived. Presto can now access data immediately after it lands in the storage layer.

Meta took two years to migrate all workloads from other interactive systems to Presto. When migrating queries to a Presto, Meta had to address the syntactic incompatibilities and implement the functions mapping between old systems and Presto, which is flexible to allow Meta to map all queries to supported Presto queries.

Because systems like Raptor, Cubrick, and Scuba load data from the lakehouse, the data migration was not a challenge, as users can use Presto to load the data from the lakehouse. At the end of the migration, Meta completely deprecated Raptor and Cubrick, saving several hundred thousand lines of code and several thousand machines.

For the batch engines, Meta also decided to migrate most of the batch pipelines to Presto.

Meta created the Hive engine for all batch processing in late 2000 and later replaced it with SparkSQL. When migrating to Presto, Meta faced a problem in which Presto’s architecture at that time was insufficiently resilient to machine failures compared to Spark when executing long-running pipelines.

They solve this problem by combining the scalability of the Spark engine with the cleaner standards-compliant SQL called PrestoSQL, which resulted in Presto on Spark. The solution achieved this by refactoring the Presto front-end (parser, analyzer, optimizer, planner) and backend (evaluation and I/O) libraries and embedding these in the Spark driver and worker.

With interactive queries already ran on Presto, Presto on Spark offers 100% compatibility with PrestoSQL; users can switch from interactive queries to batch pipelines without needing to rewrite the queries.

At Meta, Presto on Spark is currently in production and running thousands of pipelines daily.

There were also fragmented solutions in the streaming engines. The two main reasons were:

• Programming language divergence (C++ vs. Java vs. PHP).

• The abstraction level divergence (low-level procedural vs. high-level declarative API)

The legacy stacks had Puma (Java, declarative), Stylus (C++, low level), and others with different combinations of abstraction levels (declarative, procedural) and implementation languages (C++, Java, PHP).

To deal with this, Meta built XStream, the next generation of stream processing platform. Meta promoted SQL as the primary way to interact with XStream by integrating with the CoreSQL. They also made it more efficient with a Velox-based execution engine.

We will expore CoreSQL and Velox later.

XStream today supports various use cases from SQL queries and machine learning workloads to function as a service.

SQL Dialect

Meta had more than six variants of SQL being used internally. If users wanted to use different systems, there was a high chance that they had to learn a different SQL dialect. Meta decided to narrow it down to two dialects: MySQL and PrestoSQL. The first is for OLTP workloads, and the latter for OLAP workloads.

However, Meta found it challenging to achieve compatibility across the different engines. They looked around and found that the way Google achieved the same purpose with ZetaSQL could help them; they needed two components:

• The SQL parser and analyzer for parsing and analyzing queries plus creating and validating query plans. Meta already had a Java implementation (Presto) and a Python implementation (used by developer tools). They rewrote the Python implementation in C++ for better performance and better integration with the C++ engines. They are working to bind Java implementation to the C++ library.

• A library of query functions and operators. Meta initially reused the Java implementation from Presto and tried to replace it with the Velox engine to maximize the performance. We will explore Velox in the Execution Engine section.

Meta called this solution CoreSQL. It acts as the standard dialect across engines, from Presto to XStream.

Storage

Meta used ORC as the columnar format for the lakehouse. Later, they developed DWRF, the ORC variant to support better deeper nested data and finer grained encryption. Meta has fragmented codec implementation for this format: one Java implementation of Spark, one Java implementation for Presto, and one C++ implementation for ML applications.

Because of its higher performance, Meta chose the Presto codec as the base one and added necessary features to it. Then, they migrated all codecs used in Spark and other systems to the new one. In addition, Meta refactored the DWIO library into Velox, added some features, and open-sourced the library as part of Velox.

Execution Engine

Like all the areas above, the lakehouse’s evolution created fragmentation in the execution engines. More than twelve specialized engines that shared little to nothing with each other were written in different languages and developed by different teams.

To address this challenge, Meta created Velox, a C++ database acceleration library (think Databricks’s Photon in the case of Spark). Velox aimed to unify execution engines across different compute engines.

Typically, Velox receives the fully optimized query plans and performs the computation using the resources in the local machine.

As Meta claimed, Velox democratizes the optimizations that are only found in individual engines, which reduces duplication, offers reusability, and improves consistency.

At the time of the paper’s release, Meta integrated Velox into many systems. Meta also provided the implementation of the CoreSQL dialect for Velox.

Outro

Thank you for reading this far

In this article, we explored the limitations of Meta’s legacy approach for their lakehouse, how they addressed them with the Shared Foundations, and how they implemented it in different areas from the compute engine, SQL dialect, and storage format to the execution engine.

Now, it’s time to say goodbye. See you in the next articles.

Reference

[1] Biswapesh Chattopadhyay, Pedro Pedreira, Sameer Agarwal, Yutian "James" Sun, Suketu Vakharia, Peng Li, Weiran Liu, Sundaram Narayanan, Shared Foundations: Modernizing Meta’s Data Lakehouse (2023)

I’m making my life less dull by spending time learning and researching “how it works“ in the data engineering field.

Here is a place where I share everything I’ve learned. Not subscribe yet? Here you go:

Subscribe now

Intro

Nearly 50,000 companies use Apache Kafka.

Fourteen years ago, a team led by Jay Kreps built Kafka to meet LinkedIn's growing log processing demands. Since its open-source release, Kafka has become the de facto standard for distributed messaging.

But here’s the catch: Kafka’s design isn’t optimized for the cloud era. Compute and storage can’t scale independently, cross-availability-zone transfer fees due to data replication, and other challenges persist whether you run Kafka locally or in the cloud.

• Data quality concerns: Kafka brokers treat messages as raw byte sequences, leaving schema validation up to producers and consumers. If someone skips this step, downstream systems suffer.

• Pipeline complexity: Once data lands in a Kafka topic, you need an entire pipeline to move it to a data lake before analytics engines can query it.

What if there were a solution that helped you manage Kafka more efficiently in the cloud, ensured data quality, and transformed Kafka messages into an Iceberg table in just a few minutes?

Today, we explore Bufstream—the solution that promises all of this.

A bit of Kafka

Kafka achieves high throughput by leveraging a page cache and a sequential disk access pattern. It simplifies the system by relying on the OS for storage management; all read and write operations must pass through the page cache.

This tightly coupled design means that scaling storage requires adding more machines, often leading to inefficient resource usage. To address this limitation, Uber proposed Kafka Tiered Storage (KIP-405), introducing a two-tiered storage system:

• Local storage (broker disk) stores the most recent data.

• Remote storage (HDFS/S3/GCS) stores historical data.

However, brokers are not entirely stateless.

Kafka's design also relies on replication for message durability. Each Kafka partition has a single leader and zero or more followers (those storing replicas). All writes must go to the partition’s leader, and reads can be served by a leader or the partition's followers.

When the producer writes messages to the leader, the leader replicates them to followers. This helps Kafka to fail over other replicas when a broker fails. Because Kafka storage and compute are tightly coupled, any change in cluster membership forces data to move around the network.

However, Kafka's design becomes less efficient when operating in the cloud:

• It can’t fully leverage the cloud's pay-as-you-go pricing model, as computing and storage cannot be scaled independently.

• It can incur significant cross-availability-zone (AZ) data transfer fees because messages are replicated across different AZs.

That’s why many solutions are emerging to offer a cloud-native alternative to Kafka, and Bufstream stands out as a compelling contender.

Bufstream

The motivation

Bufstream was developed by Buf, a software company founded in 2020 to bring schema-driven development to the world via Protobuf and gRPC for many companies.

Protocol Buffers (Protobuf) is an efficient binary serialization format developed by Google. Unlike JSON, Protobuf enforces strict schemas using .proto files, where fields are assigned unique numbers for efficient encoding. It supports schema evolution by allowing new fields to be added without breaking existing consumers, ensuring backward and forward compatibility.

Buf has been building the Buf Schema Registry (BSR), the complete Protobuf schema registry, and a robust Protobuf package manager. As BSR grew, Buf saw more customers wanting these capabilities for data streaming use cases, specifically customers sending Protobuf payloads over Kafka.

These customers wanted to tie Kafka topics to specific Protobuf message formats, enable broker-side validation, automatically envelop raw data, and leverage BSR’s support for custom Protobuf options to enforce field-level RBAC at the gateway.

At first, Buf only built the Buf Kafka Gateway, a Kafka proxy that leveraged BSR to provide validation, automatic enveloping, and field-level RBAC. As Buf developed the gateway, object-store-based Kafka emerged (e.g., WarpStream, AutoMQ).

Realizing they could offer an end-to-end solution, they built a Kafka-compatible message queue with native support for features like directly writing Iceberg tables to S3 while bringing the same reliability and developer experience to data streaming as they did with Protobuf and gRPC.

The result was Bufstream, an enterprise-grade, object storage-based Kafka-compatible message queue verified by Jepsen.

Jepsen is the gold standard for distributed systems testing, and Bufstream is the only cloud-native Kafka implementation that has been independently tested by Jepsen.

But how is Bufstream different as a Kafka replacement?

Replacing local disks with object storage

Buf designed Bufstream from scratch to ensure 100% Kafka compatibility while storing all data in object storage. For the Kafka protocol, Bufstream supports the latest version of each Kafka API (as of Kafka 3.7.1) while striving to maintain compatibility with all previous endpoint versions.

For the storage, instead of writing to a local disk, Bufstream now writes directly to object storage like AWS S3, Google Cloud Storage, or Azure Blog Storage, allowing these services to be in charge of data durability and availability.

Unlike the tiered storage approach, which maintains local and remote storage, Bufstream stores messages entirely in the object storage. This allows users to scale computing and storage independently. Need more computing power? Add RAM and CPUs. Need more storage? Object storage enables you to expand capacity without limits (except for your budget.)

With the same setup of a single topic with 288 partitions, 1 GiB/s of symmetric reads and writes, and a 7-day data retention period on both Kafka on AWS and Bufstream, the Kafka cluster's EBS volumes cost $42,025 per month. For the Bufstream storage, it only costs $4,625 per month. The cost savings are mainly due to:

• Object storage is cheaper than disk media like AWS EBS.

• The actual data stored in Bufstream is smaller than Kafka because it doesn’t need to replicate the data between brokers.

With object storage, here is a typical message-writing process of Bufstream:

• Brokers write messages into the object storage as intake files and acknowledge the write to the producers.

• Intake files include messages from many topics and partitions and are grouped according to a time boundary.

• This message batching can help reduce the cost of writing for a single partition.

• Bufstream has a background process to organize unordered messages from intake files into archives files with the help of message-ordering metadata from the metadata store, which can be etcd, Postgres, Google Spanner,…

Reducing the cross-availability zone transfer fee

The benefit of leveraging object storage does not stop there.

According to Confluent, cross-AZ replication can account for more than 50% of total infrastructure costs when self-managing Apache Kafka, making it a significant financial consideration for cloud deployments.

In the same benchmark above, the Kafka setup requires users to pay $34,732 monthly for the cross-availability zone transfer fee, three times the cost of the Bufstream clusters.

This high cost is primarily driven by:

• Kafka producers must always write to the partition leader. If a Kafka cluster spans the leader partition into three availability zones, producers will write to a leader in a different zone approximately two-thirds of the time.

• The leader replicates the data to brokers in the other two availability zones.

With Bufstream, the cross-availability zone transfer fee is only $226 due to the metadata communication; this huge saving is mainly because:

• Bufstream stores data in object storage and lets it ensure data durability; It doesn’t need to replicate data like Kafka.

• Bufstream brokers are stateless. When adding and removing brokers, data doesn’t need to be moved over the network like Kafka. Instead, it only needs to update the metadata that maps the brokers and partitions in the object storage.

• Bufstream brokers are leaderless; any broker can serve read and write. To limit cross-availability zone traffic, Bufstream uses flags to identify the client’s availability zone (AZ) and returns only brokers within that AZ during service discovery, avoiding cross-zone data transfer.

Deployment

Deploying Bufstream is straightforward. All you need is a Helm chart, and you’re good to go; Bufstream grants the customer complete control over the deployment. While WarpStream claims data sovereignty via BYOC, they lets users secure data within their private VPC but still requires routing metadata back to the WarpStream Cloud. With Bufstream, no data is sent back to Buf. A Bufstream deployment is entirely within a customers’ VPC.

For a typical Bufstream deployment, you only need the following tech stack:

• A Kubernetes cluster

• Object storage (S3, GCS, or Azure Blob Storage)

• A metadata store (Etcd, PostgreSQL, Google Cloud Spanner, or AWS Aurora)

Pricing

So, we've explored Bufstream as a much cheaper alternative to Kafka, but how does its pricing model work?

It’s straightforward: $0.002 per uncompressed GiB written (about $2 per TiB).

Stanislav Kozlovski, a Kafka expert and writer, gives some juicy numbers for the setups to achieve 256 MiB/s throughput, 7-day retention, 4x compression, and 1GiB uncompressed:

• A Kafka setup costs $1,077,922

• A Kafka-optimized setup costs $554,958. It has tiered storage and allows the consumer to fetch data from followers to save a cross-AZ transfer fee.

• A Bufstream setup costs only $128,136, less than 8.4 times compared to the Kafka setup and less than 4.3 times compared to the optimized setup.

As Stanislav confidently said, Bufstream is the lowest-cost Kafka-replaced solution on the market.

Ensuring data quality

In addition to cutting costs, Bufstream provides first-class schema support at the broker level to help users with data quality issues. Before discovering how Bufstream can help, let’s understand how users perform data quality checks in Kafka.

Kafka sees your message as just an array of bytes. It has no clue if you tell the broker to check if a message has all expected fields or if a field has a string value instead of an integer.

The schema validation process must occur outside the brokers with the help of the Schema Registry, a centralized service that manages and enforces data schemas for Kafka topics, ensuring consistency and compatibility between producers and consumers.

The Schema Registry operates independently of the Kafka brokers and interacts with producers and consumers through a RESTful API. The topic schemas are stored and referenced by unique schema IDs. A typical process is:

• The producer has two client instances, one for the Kafka cluster and another for the Schema Registry.

• The producer checks whether the schema is already in the Schema Registry. If it doesn't, the producer sends a POST request to register it.

• The producer retrieves the schema ID from the Schema Registry.

• The producer serializes the message with the schema ID and sends the serialized message to the Kafka broker.

• The consumer also has two client instances, one for the Kafka cluster and another for the Schema Registry.

• The consumer polls the Kafka broker for new messages.

• It extracts the schema ID from the first few bytes of the serialized message.

• It then sends a GET request to the Schema Registry, using the schema ID, to retrieve the schema.

• The consumer deserializes the message according to the schema, converting the binary data to its original format.

There are some problems:

• A misconfigured producer can send malformed or unregistered messages.

• Bad data can still enter the system if a producer forgets to validate the schema.

• The producer and consumer clients become thick. They must handle the schema validation logic, which increases code complexity, dependency management issues, and inconsistency across teams.

Bufstream takes a different approach when treating schema as the first-class citizen with Protobuf messages in both the binary format and the ProtoJSON format. Buf is working to support Avro and JSON messages in the future.

The broker can check and reject messages that don't match the topic's schema. It achieves this by integrating with any schema registry that implements the Confluent Schema Registry API, including the Confluent Schema Registry itself and the Buf Schema Registry (BSR). This Schema Registry serves as a single source of truth for all the Protobuf assets, including the .proto files that define the data schema.

Whenever the Bufstream broker receives a Protobuf message from the producer, it checks whether the message’s schema matches the topic schema defined in the BSR. If yes, the broker accepts the message and prepares for the upcoming write to the object storage. If not, it rejects the message and informs the producer.

In Kafka, client-side validation isn’t really validation; clients opt-in to do that. A trusted, centralized validation point is needed, which, in this case, is the broker. Since all clients connect to the broker, validation can be enforced there. Relying on client-side validation is risky because clients can simply skip it.

Additionally, Bufstream can offer a more robust way to ensure data quality; although the schema validation process can help prevent bad data, it is sometimes insufficient.

• You expect the “age” field to be an integer, but what if the field with 350 arrives?

• You expect the “email“ field to be a string, but what if the field “abc“ arrives?

Schema validation can not find the unusual here. Bufstream lets you implement semantic validation of Protobuf messages at runtime based on user-defined validation rules. For example, an age field must have a value from 0 to 120, or an email must have an “@.”

Beyond data quality, Bufstream provides granular access control, allowing on-the-fly Protobuf redaction and exposing some fields to consumers. Currently, this logic is static, but Buf plans to introduce field-level RBAC, enabling producers to tag sensitive fields in Protobuf schemas; consumers will only receive authorized data.

Kafka topic → Iceberg table

Suppose we want to execute analytics on Kafka messages, such as ad-hoc exploration or reporting. We must build a pipeline with Kafka Connect, Spark, or Flink to consume messages from the Kafka topic, write them into files, and push these files to the data lake.

We have to take care of everything from managing the pipeline to ensuring the physical layout of the files is optimized for downstream consumption (e.g., too many small files can hurt the read operations)

Since Bufstream already stores the topic’s messages in object storage, it transforms data in transit to rest in S3 as Parquet files with Iceberg metadata on top. Users don’t have to deploy, monitor, or manage a dedicated data pipeline. Bufstream will handle all that. With schema awareness, Bufstream can synchronously update the user's iceberg catalog to notify them of field changes or new files.

Here’s an interesting point: the way Bufstream stores the Iceberg table is very unique. Other systems, such as Tableflow from Confluent, promise to write Kafka messages to an Iceberg table by reading Kafka data and copying it over, thereby duplicating data for two different purposes—serving consumers and handling analytics workloads. In contrast, Bufstream only stores the Iceberg tables.

Remember the Bufstream message-writing process mentioned above? Initially, it writes messages into intake files and later rewrites them into archive files. With Iceberg integrations, Bufstream will rewrite the intake files directly into Iceberg tables. It uses the Iceberg table for both Kafka and the lakehouse storage layer. The query engine can tap into this layer to process data, while the broker will read data from these Iceberg tables and return it row by row to consumers when they poll for messages.

Using the Iceberg table as a 2-for-1 solution like this can lead to massive storage savings. With this Bufstream feature, users can reuse the storage already allocated for Iceberg tables in the lakehouse, effectively eliminating the cost of Kafka storage altogether.

With support for popular Iceberg catalogs like REST Catalog, BigQuery Metastore, and upcoming support for Databricks Unity Catalog, Snowflake Polaris, and AWS Glue, you can seamlessly use any Iceberg-compatible query engine to access Iceberg tables from Bufstream.

Here is a process of transforming Kafka messages into an Iceberg table from Bufstream:

• With Iceberg integration, the user needs to specify the archive format is iceberg.

• First, the writer contacts the BSR to fetch the latest schema and caches it in memory for later use with the same topic messages.

• The writer uses this schema to form the Iceberg table schema. To handle schema evolution, Bufstream keeps the Iceberg schema state in the metadata store.

• After the writer forms the schema, it talks to the Iceberg catalog to check if it has changed. If yes, the writer updates the schema in the metadata store. If the destination table does not exist, the writer creates the table and sets the schema ID to 0.

• The writer derives the Parquet schema from the Iceberg schema to prepare to write the data files.

• After writing the Parquet data files, the writer writes the manifest files, the manifest lists, and the metadata files.

• Finally, the writer tells the catalog to update the table’s current metadata pointer to the new metadata file.

My Thoughts

Although choosing to store data in object storage can make Bufstream way cheaper than Kafka, it must sacrifice the low-latency performance of the disks. In their benchmark, the median end-to-end latency was 260 milliseconds, and the p99 latency was 500 milliseconds. Still, these numbers are considerably better than those of other solutions, such as WarpStream.

Bufstream offers a way to optimize latency. It batches messages before writing to object storage to limit the PUT request. Thus, users can adjust the batch size to reduce latency, but more frequent PUT requests to object storage will increase the cost.

Given the vast cost savings compared to Kafka, Bufstream’s latency is acceptable. Unless you’re dealing with use cases that require super low latency, Bufstream's latency sacrifice does not impact much.

But if we set latency aside, Bufstream presents a strong alternative to Kafka in the cloud. Beyond cost efficiency, it offers a straightforward deployment model, built-in schema awareness for data quality enforcement, and the seamless transformation of Kafka’s storage layer into a lakehouse.

The native Iceberg support is a very valuable feature for me. In data engineering, transforming message queue data into analytics tables is inevitable. By transforming Kafka topics into Iceberg tables, Bufstream significantly reduces the burden on data engineers. The Iceberg format ensures broad compatibility, letting us use our favorite query engine over it, from Databricks, Snowflake, and BigQuery to Spark or Trino. Avoiding vendor lock-in is a big win for any company.

Outro

Thank you for reading this far!

Throughout this article, we’ve explored why Kafka may be inefficient in the cloud, how Bufstream offers a more cost-effective alternative by storing data in object storage, how it enhances data quality by making the broker schema-aware, and how Bufstream seamlessly transforms topic messages into Iceberg tables. We wrap up the article with some of my naive thoughts.

Now it’s time to say goodbye. See you in my next articles :)

Reference

[1] Gwen Shapira, Todd Palino, Rajini Sivaram, Krit Petty, Kafka The Definitive Guide Real-Time Data and Stream Processing at Scale (2021)

[2] Confluent, Schema Registry Clients in Action (2024)

[3] Bufstream Documents

[4] Stanislav Kozlovski’s Bustream post on Linkedin

I’m making my life less dull by spending time learning and researching “how it works“ in the data engineering field.

Here is a place where I share everything I’ve learned. Not subscribe yet? Here you go:

Subscribe now

Intro

AWS Lambda is a fascinating service.

I first used it in 2021, and the experience was seamless. I wrote some code and configured how it should be triggered, and that was it.

Whenever a new file arrived in S3, my Lambda function would wake up, execute some logic, and then go back to sleep. I didn’t have to worry anything about the infrastructure.

That made me wonder: could I achieve the same simplicity with my data pipelines?

What if there was no need to set up an Airflow environment or provision a Spark cluster? What if I could define the pipeline logic—similar to an AWS Lambda function—and somehow, the input data would transform into the desired output?

This week, we’re diving into Bauplan, a solution that makes that wish come true.

Overview

Function-as-a-Service (FaaS) is a cloud computing model that allows developers to run code in response to events without managing the infrastructure. It enables a serverless approach, where the cloud provider handles provisioning, scaling for bursty workloads, and execution, allowing engineers to focus on writing logic. A well-known example is AWS Lambda.

FaaS makes it simple for the developer.

Bauplan, a team from New York and San Francisco, believes that the FaaS model can also simplify work for data engineers, analysts, data scientists, or anyone who wants to work with data.

Still, the solutions available on the market cannot adapt to the data workload. We usually define a data pipeline as a Directed Acyclic Graph (DAG), in which each node is a function that receives data from previous nodes, applies logic, and outputs results for the following nodes.

Modularizing the business logic into nodes makes developing, collaborating, and testing more convenient. But when implementing the DAG data pipeline using available FaaS solutions, some challenges emerge:

• Scaling: Existing FaaS runtimes are designed for simple, independent functions that produce small outputs (e.g., a webhook). They have limitations when applied to data pipelines. Additionally, these FaaS platforms usually reuse instances for subsequent triggers. It can use the same function instance for 10 GB and 1TB of input with the same data function. “Out of memory” errors are common.

  

• Large intermediate I/O: To implement the DAG concept, users must chain functions. A function acts as a “node“ that receives input from previous functions and produces output for the following functions. Data functions typically have large inputs and outputs, which increases the cost of serializing and moving the data payload between functions. Popular FaaS platforms' chaining best practices are limiting because intermediate data frames can only be transferred through object storage.

  

• Slow feedback loop: Data science projects are exploratory and require rapid iteration to validate hypotheses. Current FaaS platforms lack the interactivity needed for these projects due to their slow build times and lack of interactive logging. AWS Lambda provides only observability through Cloudwatch.

  

So, how does Bauplan promise to solve these problems?

The Bauplan FaaS

Bauplan is a FaaS service designed for data pipelines. Unlike other services, Bauplan initiates and scales independent instances for every run. It also promises to boost the intermediate data exchange process and allow users to modify and run DAGs interactively.

For the design principles:

• Bauplan aims to make an execution stateless, and its instances only live during that run. Starting with new instances each time enables Bauplan to adapt to different resource requirements; the same pipeline can run with a 10GB dataset and later scale up to a 100 GB dataset.

• For the infrastructure, the Bauplan pipeline will run on the cloud Virtual Machines (VMs), which offer the highest level of customization. Using cloud VMs also allows Bauplan to offer multiple deployment models, such as BYOC, where customers can control where data is stored and processed.

• Bauplan is different from other tools because Bauplan has both data and runtime awareness (i.e., serverless runtimes like AWS Lambda don’t know about the data, and orchestration tools don’t know about the runtime). We’ll explore this design more deeply when we run some code later.

• Bauplan brings an interactive experience to the developer; although the pipeline runs on the cloud, users can develop as it runs on their laptops. Bauplan provides a CLI tool and Python SDK for users to interact with the system.

• Users define a function in Bauplan by specifying tables as input and output.

Architecture

Bauplan has a Control Plane (CP) and a Data Plane (DP):

• The CP exposes multi-tenant APIs. It only deals with metadata. The CP lives in Bauplan’s VPC.

• Each customer has a DP, a fleet of one or more cloud VMs that can be deployed in the customer VPCs. A Golang binary is installed in each VM to spawn the worker. These workers are the only Bauplan components that can access customer data.

To enable the developer to have an interactive experience, there is a bidirectional gRPC connection between customers and workers. Users write some print or logging statements to understand what happens during the pipeline run; although the code is run inside cloud VMs, the results are immediately visible to users thanks to the bidirectional connection.

Planning

So, the CP needs to deal with metadata, but what is its responsibility?

Bauplan acts like a database; it translates Python and SQL code into an execution plan when it begins running the pipeline. When the user requests a run, the code is routed to the control plane (CP). The CP will parse the code and reconstruct the DAG topology from the functions, resulting in a logical plan.

This plan only represents the dependencies between steps and the required packages as specified by the user. Importantly, Bauplan will refuse to run DAGs that refer to non-existing tables (unlike dbt, for example), point to wrong snapshots, or ship Python code with invalid formatting.

To give running instructions to the worker, the CP forms the physical plan from the logical one. This physical plan contains instructions for the containerized runtime of the transformation functions and mapping dataframes to a physical table in object storage. When having the physical plan, the CP sends it to workers to start the execution.

Data in Bauplan are store in Iceberg table in object storage, we will explore the storage layer soon.

The cache

As mentioned, function instances only exist during execution time. The two runs of the data pipeline will have different sets of instances. To reduce the latency when re-running the pipeline, Bauplan developed a robust package caching mechanism that avoids re-installing packages across runs, thus avoiding the overhead calls to PyPI.

For data caching, Bauplan’s data awareness makes database-like optimizations possible:

• Re-using intermediate data: Functions produce intermediate dataframes, and Bauplan tracks the change in code and data to cache and reuse intermediate data.

  

• Retrieving only missing columns: The first run reads four columns from the table, and the second run requires exactly these four columns plus column X. Bauplan will reuse the four columns from the cache and only download one additional column X from the data source.

  

• Cache invalidation: Because the physical data are stored in immutable files (via the Iceberg metadata), dataframe changes are identified with data commits such that the cache knows when data needs an update.

  

The data exchange

To enhance the data exchange process between functions, Bauplan represents intermediate dataframes as Arrow tables. From the official document:

The Arrow columnar format includes a language-agnostic in-memory data structure specification, metadata serialization, and a protocol for serialization and generic data transport.

Unlike file formats like Parquet or CSV, which specify how data is organized on disk, Arrow focuses on how data is organized in memory.

Arrow store values for each column contiguously in memory. This design is highly advantageous for data analytics workloads, which focus on a subset of columns when dealing with large datasets.

Before Arrow, each system used its internal memory format. When two systems communicate, each converts its data into a standard format before transferring it, incurring serialization and deserialization costs. Apache Arrow aims to provide a highly efficient format for processing within a single system. As more systems adopt it, they can share data at a very low cost, potentially even through shared memory at zero cost.

When Bauplan executes the pipeline, it will pick the sharing mechanism: memory or local disk (functions in the same worker) or Arrow Flight (across workers). Because other solutions only support S3-backed data exchange, moving data between functions in Bauplan can be hundreds of times faster, thanks to Arrow.

With Arrow, functions can read tables from shared memory, memory-map, or stream them from gRPC (with Flight), which gives the function greater flexibility when dealing with multiple data sources with different data transfer mechanisms.

A memory-mapped file is a segment of virtual memory that has been assigned a direct byte-for-byte correlation with some portion of a file or file-like resource. The benefit of memory mapping a file is increasing I/O performance, especially when used on large files.

Moreover, if a downstream function runs on the same worker as the upstream function, it can read the Arrow intermediate data on the worker without copying the data. Given intermediate data with 10GBs and four functions needed to read it, it only takes 10Gbs physical RAM instead of 10x4=40Gbs.

The storage

Bauplan does not stop there; beyond the FaaS data pipeline, they also aim to provide a complete lakehouse solution by offering a storage layer with Iceberg and Project Nessie.

If you have some Parquet files in the object storage, Bauplan can help you transform them into an Iceberg table in a single line of code. The data stays in your VPC; it doesn’t need to move anywhere.

Netflix created Apache Iceberg to achieve better table correctness and faster query planning (than Hives). An Apache Iceberg table has three layers organized hierarchically:

• The data layer stores the table’s actual data, including the data and deleted files.

• Manifest files track the data files in the data layer.

• A manifest list captures the snapshot of an Iceberg table at a specific moment.

• Metadata files contain information about an Iceberg table at a specific time, such as the schema or the latest snapshot.

• The catalog is where every Iceberg data operation begins. It provides the engine with the location of the current metadata pointer and tells you where to go first.

Like other table formats, Iceberg's ultimate goal is to bring data warehouse capabilities to the data lake; one important one is the ACID constraints.

The Iceberg only ensures atomic transactions at the table level. To bring the software development experience to the lakehouse, Bauplan uses Project Nessie for the Iceberg table catalog. It is an open-source versioned metadata catalog that enables cross-table transactions for Iceberg. Users can update multiple tables together and guarantee all changes occur atomically – an all-or-nothing commit across tables.

Bauplan makes it easier for users to seamlessly work with Nessie and Iceberg tables by providing CLI commands and Python SDKs. Users will feel just like they are working with a Git repository.

In the next section, we'll explore all the cool Bauplan features mentioned above, where we'll run some code.

Run some code

We will run some Python code and CLI commands; I prepared a Git repo so you can follow along. Make sure you pull the repo locally and enter the bauplan_example folder.

First, we need to set up a Python virtual environment with the requirements.txt file from the repo. We will install bauplan, streamlit and duckdb packages.

Next, we need the Bauplan API key, which gives you access to the Bauplan sandbox environment. You can contact Bauplan here for the key.

bauplan config set api_key "your_bauplan_key"

Then, we will run some bash scripts to set up; let’s make those scripts executable:

chmod -R +x scripts/

Bauplan is designed to operate exclusively in the cloud to ensure a fully auditable and secure data development cycle. They require us to store data in object storage so it can support importing to the Iceberg table.

Currently, Bauplan only supports S3 as the data source and Parquet and CSV as file formats. We will run a script that creates an S3 bucket, uploads some CSV files, checks out to a branch, creates a namespace, and then imports these files to the Iceberg table in Bauplan Sandbox. But first, make sure you configure your AWS CLI:

aws configure # Entering AWS access key, secret and default region

Then, run the setup.sh with the S3 bucket name you want to create and the Bauplan branch. The branch must be in a pattern <your-user-name>.<something>

./scripts/setup.sh <bucket name> <bauplan branch>

Wait for a while, and then you will have five tables in your catalog. The script will create a namespace called adventure. A namespace in Bauplan is a logical container that helps organize tables within a data catalog. We will work on your input branch and the adventure namespace from this time. Let’s list out the input tables:

bauplan table --namespace adventure

Before we move on, let’s understand the input data. We will have five input tables from the AdventureWorks sample dataset: product, product_category, product_subcategory, sale, and territories.

AdventureWorks database supports standard online transaction processing scenarios for a fictitious bicycle manufacturer - Adventure Works Cycles.

The relationship of the tables is:

In this project, we will write a Bauplan pipeline to transform these input tables into a dimensional data model with a fact_sale, dim_product, and dim_country:

We implement the transformation pipeline using Bauplan models. A model function takes tabular data as input and produces tabular data. We will write a DAG to transform the data using the duckdb engine:

We define these models in the models.py file. A very important point is that the Bauplan model is aware of both data and runtime.

For runtime awareness, you can use Bauplan’s decorator to specify the runtime for each model, such as the Python version and packages, how to materialize the output, and allow for explicit column selection and filter pushdown.

For data awareness, each model must have inputs, which can be tables in the catalog or other models. We specify and use them like Python function parameters.

Here is the code of the dim_product model. As you can see, for this model, Bauplan knows that it must run the model with Python 3.11, duckdb 1.0.0 and the model has product, product_category, product_subcategory as input data.

You can check the codes for all the models here.

After having the models, we run the pipeline:

bauplan run --project-dir pipeline --namespace adventure 

We submit the code to Bauplan. It will plan and execute it. If there are any errors, it will display in real-time in the terminal for us thanks to the bidirectional gRPC connection between us and the Bauplan workers.

To run the pipeline, we need a bauplan_project.yml file containing the project’s unique ID and name. I located both the bauplan_project.yml and models.py files in the pipeline folder.

After the pipeline finishes, you can check the output tables by listing the namespace adventure again:

bauplan table --namespace adventure

Finally, to have more fun with the project, I created a small Streamlit app with a world-class SQL editor (:D) to query the output table:

streamlit run streamlit/app.py 

Here is a quick demo video to showcase my world-class SQL editor:

To clean up, you can run the clean_up.sh to clean up the S3 bucket and Bauplan table automatically:

./scripts/clean_up.sh <bucket name> <bauplan branch>

My thoughts

In a world where data is the new gold, every company wants the ability to capture, store, process, and serve data to drive business decisions. However, not every company has a dedicated data team. In many cases, you might be the team's first and only data person.

At the beginning of this article, I had a wish—and it came true. Bauplan handles tasks that would typically require an entire infrastructure team. Its goal is to provide a seamless, developer-friendly way to work with large-scale data directly in Python, eliminating infrastructure bottlenecks.

When running some code with Bauplan, I was truly impressed by how seamlessly it imports data files from object storage into Iceberg tables. Setting up an Iceberg catalog, configuring the Iceberg writer, and managing the physical layout is usually a painful process, but Bauplan simplifies it significantly.

Defining data transformations is also a pleasant experience, thanks to Bauplan’s concept of a “model.” I can run transformations with Python 3.9 or 3.10 simply by changing a few lines in a decorator. The model’s data-awareness makes it incredibly intuitive to write transformation logic—I can specify it as easily as defining function inputs in Python.

Bauplan is truly innovative. It’s well worth your time trying, especially if you’re a data engineer, data analyst, or data scientist—or if you simply love working with data.

Personally, I hope Bauplan will expand to support data processing runtimes like Spark or Trino. A serverless Spark or Trino cluster would be a game-changer. Additionally, a robust SQL editor for querying data in the catalog would be a valuable addition.

Outro

Thank you for reading this far.

In this article, we explore the challenges of implementing the data pipeline with available FaaS solutions, how Bauplan promises to solve them, Bauplan’s design goals and architecture, how Bauplan offers a complete zero-infrastructure lakehouse with the Iceberg + Project Nessie storage layer, and finally we write some code to build a Bauplan pipeline.

Now, it’s time to say goodbye. See you in my following articles.

Reference

[1] Bauplan Documents

[2] Jacopo Tagliabue, Tyler Caraza-Harter, Ciro Greco, Bauplan: zero-copy, scale-up FaaS for data pipelines (2024)

To celebrate Lunar New Year (the true New Year holiday in Vietnam), I’m offering 50% off the annual subscription. The offer ends soon; grab it now to get full access to nearly 200 high-quality data engineering articles.

50% off the annual subscription

Intro

Today, we will explore how Airbnb builds and serves its semantic layer internally and what we can learn from it. More correctly, Airbnb did not only build a layer that “simplifies interactions between complex data storage systems and business users.“ They create a complete platform.

Motivation

In 2010, Airbnb had only one data analyst. His laptop was Airbnb data warehouse. He often ran queries right on production databases, and Airbnb.com was down for some time because of the heavy queries.

In the early 2010s, they hired more data scientists, and data kept growing. They upgraded their data infrastructure and built their own data orchestration tool, Airflow, with later open source.

Their top priority was to build a set of tables called core_data. These tables set the foundation for many data demands at Airbnb:

• Airbnb’s experimentation platform for streamlining the A/B testing processes.

• Dataportal — Airbnb's internal data catalog organizes and documents data assets.

• Interactively data exploration with Apache Superset

• Data University — a program that teaches non-data scientists valuable skills to democratize data analysis at Airbnb.

However, the growth came with challenges:

• More users wanted to consume core_data, so they created many tables on top of it. There was no way to tell if a table with the exact requirement existed.

  

• Because of the complexity of the growing warehouse, Airbnb found it challenging to track data. Data users could spend many hours debugging the data discrepancies.

  

• For data consumption, decision-makers complained that different teams reported different numbers for simple business questions. As a result, business users and even data scientists lost trust in the data.

  

To celebrate Lunar New Year (the true New Year holiday in Vietnam), I’m offering 50% off the annual subscription. The offer ends soon; grab it now to get full access to nearly 200 high-quality data engineering articles.

50% off the annual subscription

Airbnb Minerva

They revamped the data warehouse to improve data quality.

First, their data engineering team rebuilt key business data models, resulting in lean tables that eliminate redundant joins. These tables served as the new foundation for the analytics.

That still was not enough.

They needed to join these tables to extract insight, backfill data whenever logic changes, or present the data consistently and correctly in many different consumption tools.

Airbnb built Minerva for these purposes.

Minerva took fact and dimension tables as inputs, performed data denormalization, and served the aggregated data to downstream applications. Airbnb hoped the Minerva API would close the gap between upstream data and downstream consumption.

At the time of Airbnb’s sharing, Minerva contained more than 12,000 metrics and 4,000 dimensions, with 200+ data producers across different functions and teams.

Architecture

Airbnb built Minerva on top of open-source projects:

• Airflow for workflow orchestration.

• Apache Hive and Apache Spark for the compute engine.

• Presto and Apache Druid for serving.

For a metric, Minerva has components to cover its whole life cycle:

• Minerva defines metrics, dimensions, and metadata in a centralized Github repository. Anyone at Airbnb with proper permissions can update these definitions.

• It has a development flow for code review, static validation, and test runs.

• It executes data aggregation/denormalization by resue data assets and intermediate joined results.

• Minerva has a robust computation flow that can automatically retry after job failures, plus the built-in data-quality checks.

• It exposes a unified data API to serve metrics and metadata.

• Because the Minerva version controls data definitions (via Git), it can detect and track changes and then execute data backfilling.

• Its data management features include cost attribution, GDPR-based deletion, or data access control.

• For data retention, Minerva supports clean-up of data based on usage; infrequently used datasets can be deleted to save cost.

Design principle

Airbnb built Minerva to be:

• Standardized: Data is defined in a single place. It must serve as a single entry point for anyone searching for definitions.

• Declarative: Users define the output they want (like SQL). Minerva will handle everything from calculating metrics to storing and serving.

• Scalable: Minerva must be scalable to support Airbnb’s internal data demands.

• Consistent: The data is always consistent. If the user changes the definitions, Minerva must perform data backfill and keep the data up-to-date.

• Highly available: Dataset replacement must be handled with minimal impact on data consumption.

• Well-tested: Users can prototype and validate their changes before they are merged into production.

Standardized

Minerva is focused on metrics and dimensions instead of tables and columns like databases.

When a metric is defined in Minerva, users must provide necessary metadata, such as ownership, lineage, or metric description. Before Minerva, Airbnb managed metadata inefficiently as definitions scattered across various business intelligence tools.

Regarding version control in Minerva, they treat all definitions as code that must go through a review process before merging to production, just like code review.

Minerva’s configuration system cores are event and dimension sources, corresponding to fact tables and dimension tables in the data warehouse:

• Event sources define the atomic events which are used to calculate metrics.

• Dimension sources contain attributes that can be used with the metrics.

Declarative

One of Minerva’s promises is to simplify the time-consuming workflow so that users can quickly turn data into insights. Users can define a dimension set, an analysis-friendly dataset created from Minerva metrics and dimensions. Unlike ad-hoc datasets, dimension sets have several advantages:

• Users define what they want. Minerva abstracts all the technical implementation details and complexity of creating it from the users.

• Dimension sets can benefit from Minerva’s existing features.

• Minerva can store and optimize these dimension sets to reduce query times.

• Minerva can reuse dimension sets, which help reduce dataset duplication.

Scalable

Minerva was serving 5,000+ datasets across hundreds of users and 80+ teams.

To ensure it can scale, Airbnb built Minerva’s computation with the DRY (Do not Repeat Yourself) principle. They tried to reuse materialized data as much as possible to reduce wasted computing resources.

The computational flow has several stages:

• Ingestion Stage: Minerva sensors are triggered when new data is added to the table’s partitions. The latest data is then ingested into Minerva.

• Data Check Stage: This stage ensures that upstream data is “right. “ For example, a field should not be empty, or primary keys should be unique.

• Join Stage: Minerva executes the joins based on join keys to generate dimension sets. Minerva computes the same calculations (e.g., same city dimension) that happen on different dimension sets using the same logic on the same source tables. This ensures consistent dataset computation at scale.

• Post-processing and serving stage: Minerva further aggregates outputs for downstream consumption. It can optionally optimize data end-user query performance.

In addition, Airbnb included features to make Minerva operate efficiently. Some features are self-healing and automated backfilling.

Minerva tries to be data-aware. It checks for missing data for every job. If missing data is identified, it is included in the current run. This means a single run can have a data range changed dynamically (e.g., 3 days → 4 days of data). Users don’t need to reset tasks manually.

For the backfilling, if the backfill window is long (e.g., several months), it may generate an expensive query. If they split the backfill window into smaller ones, it will be too slow for a large initial window. To solve this, Airbnb introduced the batch backfill for Minerva.

They still split the backfill window into smaller ones based on the scalability of that dataset. For example, a one-year window would be divided into 12 1-month windows. Then, they run these 12 jobs in parallel.

Consistent

Internal users frequently change Minerva's definitions. Airbnb introduced a data version to ensure that Minerva datasets are consistent and up-to-date.

The data version is a hash of all the essential fields specified in the definitions (e.g., data source). When users change any field used for the hashing, the data version is automatically updated.

Each dataset has a data version, which makes Minerva automatically create and backfill a new dataset. This approach ensures that upstream changes are propagated to all downstream datasets, and no Minerva dataset will diverge from the source of truth.

Highly Available

Airbnb observed that backfills often could not catch up with user changes when updates affect many datasets. Given that Minerva promises to provide consistent and up-to-date data, a frequently changing dataset could result in backfill forever and cause data downtime.

Airbnb deployed a parallel computation environment called the Staging environment. The Staging environment replicates the Production environment. They will perform data backfilling in the staging before publishing it on the Production. The flow for the Staging environment is as follows:

1. Users developed and tested changes in the local environment.

2. They merge changes to the Staging environment.

3. The Staging environment loads the Staging configurations, retrieves any necessary Production configurations if needed, and starts backfilling modified datasets.

4. The Staging changes are merged into Production when the backfill is done.

Well-Tested

To help users validate data correctness, Minerva has a tool that reads from production but writes to a sandbox environment. The tool generates sample data on top of the user’s local modifications, allowing users to validate their changes.

The tool shows the step-by-step computation that Minerva follows to generate the output. This feature helps users debug issues just like they are running the logic. Finally, it also allows users to configure date ranges to limit the test data size, which helps them save a lot of time waiting for the test to finish.

Consumption

The Minerva teams partnered with other internal teams to create an ecosystem around Minerva:

• Data catalog: They index all Minerva metrics and dimensions in Airbnb’s Dataportal. When a user searches for a metric, the Dataportal shows the result from Minerva.

• Dataportal also offers a data exploration feature called Metric Explorer. Users can see metric trends with additional slicing and drill-down options, such as Group By and Filter. Users who want to dig deeper can switch to Superset to perform more advanced analytics.

• They migrate the A/B test platform’s proprietary metric repo to Minerva, which helps achieve consistency across experimentation and analytics.

• To enable executive reporting, they built a reporting framework that turns a set of user-specified Minerva metrics and dimensions into aggregated metric time series.

• Minerva exposes API for Airbnb’s R and Python clients. This lets data scientists query Minerva data in a notebook environment. Data scientists can now have metric calculation results exactly like those of other tools such as Metric Explorer, saving them lots of time when investigating data discrepancies.

Outro

Thank you for reading this far.

In this article, we explore the motivation behind the need for the semantic platform from Airbnb, the platform architecture and design principle, and finally, how Minerva can serve downstream consumption.

Now it’s time to say goodbye. See you in my following articles.

Reference

[1] The Airbnb Tech Blog, How Airbnb Achieved Metric Consistency at Scale (2021)

[2] The Airbnb Tech Blog, How Airbnb Standardized Metric Computation at Scale (2021)

To celebrate Lunar New Year (the true New Year holiday in Vietnam), I’m offering 50% off the annual subscription. The offer ends soon; grab it now to get full access to nearly 200 high-quality data engineering articles.

50% off the annual subscription

Intro

This week, we will explore the differences between open-source Spark and Databricks Spark, why the creators originally developed Spark, why Spark alone is insufficient for Databricks' Lakehouse solution, and how Databricks makes Spark significantly more efficient.

Apache Spark

Why it was created

Apache Spark is an open-source distributed computing system designed to quickly process large volumes of data that can hardly accomplished by operating on a single machine. Spark distributes data and computations across multiple machines.

It was first developed at UC Berkeley’s AMPLab in 2009.

At the time, Hadoop MapReduce was the popular choice for processing big datasets across multiple machines. AMPLab collaborated with early MapReduce users to identify its strengths and limitations. They also worked closely with Hadoop users at UC Berkeley, who focused on large-scale machine learning requiring iterative algorithms and multiple data passes.

These discussions highlighted some insights. Cluster computing had significant potential. However, MapReduce made building large applications inefficient, especially for machine learning tasks requiring multiple data passes. For example, the machine learning algorithm might need to make many passes over the data. With MapReduce, each pass must be written as a separate job and launched individually on the cluster.

To address this, the Spark team created a functional programming-based API to simplify multistep applications and developed a new engine for efficient in-memory data sharing across computation steps.

SparkSQL

Spark was intended to focus more on a general-purpose cluster computing engine than a specified database’s query engine. Realizing the need for relation processing over big datasets, the people behind Apache Spark presented the new model Spark SQL in 2014. This new module lets Spark programmers leverage the benefits of relational processing (e.g., declarative queries and optimized storage). Spark SQL introduces two significant enhancements.

• First, it integrates relational and procedural processing through a declarative DataFrame API.

• Second, it incorporates a highly extensible optimizer, Catalyst, which leverages Scala's features to facilitate the addition of composable rules and manage code generation.

The goals of SparkSQL are:

• Support relational processing of Spark’s native RDDs and external data sources using a convenient API.

• Offering high performance using DBMS techniques.

• Efficiently supporting new data sources,

• Enabling extension with advanced analytics algorithms such as graph processing and machine learning.

The people behind Spark aim to make it a viable option as a query engine.

Databricks

The Apache Spark team founded Databricks in 2013. The company aims to simplify the process of building and deploying Spark applications for organizations. In 2019, Databricks introduced Delta Lake, a table format that provides the warehouse capability to the data lakes.

In 2021, they released a paper introducing the new data management paradigm, the Lakehouse. This paradigm combines the best of both worlds: the warehouse's robust management features with the lake's theoretically unlimited scalability.

Databricks aimed to solve some problems with the two-tier data architecture, such as the stale data in the warehouse compared to the lake’s, the difficulty and cost of consolidating the data lake and warehouse, and users being billed twice the storage cost for data duplication in the data lake and warehouse.

They have been offering the managed lakehouse solution with Delta Lake for the storage layer and Spark for the query engine.

The challenges

Databricks does not just want to offer a data management system; it must also ensure high performance to compete with other solutions in the market, such as Snowflake, BigQuery, and Redshift.

At that time, all the above solutions primarily positioned themselves as cloud data warehouse solutions—the lakehouse paradigm caused Databricks some problems because Spark was initially not developed to be a native query engine:

• The Lakehouse query engines deal with a greater variety of data than traditional warehouses. From organized datasets to raw data with messy layouts, many small files, many columns, and no valuable statistics, the execution engine must be flexible enough to deliver good performance on a wide range of data.

• Databricks initially offered Spark as the lakehouse engine. To enhance the query engine, they must ensure that many customers using Spark do not experience disruptions.

They need a more efficient query engine but can’t replace Spark. So, what did they do? Simple—they enhanced Spark in place.

To celebrate Lunar New Year (the true New Year holiday in Vietnam), I’m offering 50% off the annual subscription. The offer ends soon; grab it now to get full access to nearly 200 high-quality data engineering articles.

50% off the annual subscription

Their effort

An important thing to note is that before this effort to enhance Apache Spark, Databricks already built their own Spark runtime, the Databricks Runtime (DBR), which is a fork of Apache Spark that provides the same interface but has enhancements for reliability and performance.

But they need a little more than that.

They built the Photon engine, a library that integrates closely with the DBR. The engine acts as a new set of physical operators inside the DBR. The query plan can use these operators like any other Spark. Databricks’s customers can continue to run their workloads without any changes and still benefit from Photon.

The system can run the queries partially in Photon; if it needs unsupported operations, they are switched back to SparkSQL. Databricks tests Photon to ensure its semantics are consistent with Spark SQL’s

Databricks built Photon using a vectorized model instead of the code generation approach that Apache Spark implements. Vectorized execution enabled support runtime adaptivity; Photon discovers, maintains, and leverages micro-batch data characteristics with specialized code paths to adapt to the properties of Lakehouse data.

Another essential design that Databricks made when developing Photon is writing it in C++ instead of following the Spark approach, which used the Java Virtual Machine (JVM). Databricks observed that “the Spark applications were hitting performance ceilings with the existing JVM-based engine.” Moreover, they found that the performance of native code was more effortless to explain than that of the JVM engine, as they can explicitly control aspects like memory management and SIMD in C++.

The Photon Designs

JVM vs. Native Execution

Databricks decided to move away from the JVM and implement a native code execution engine. Integrating the new engine with the existing JVM-based runtime is challenging for Databricks. Here are several reasons that led Databricks to the decision to develop a new native execution engine:

• The Lakehouse paradigm demands processing a wide range of workloads that stresses the JVM engine's in-memory performance.

• Improving the JVM engine performance requires deep knowledge of JVM internals.

• Databricks found they lack control over lower-level optimizations such as custom SIMD kernels.

• They also observed that garbage collection performance degraded on heap memory larger than 64GB. Databricks had to manually manage off-heap memory in the JVM-based engine, which made the codebase more complex.