Not subscribed yet? Here you go:

Subscribe now

👋 Hi, my name is Vu Trinh, a data engineer.

I share my lesson and excellent resources to read in this newsletter.

Hope this issue finds you well.

Meta: Overview of the internal data stack

Intro

A planet-scale company like Meta, Twitter, or LinkedIn processes tons of data daily to support its operations. To support that need, those companies must build a robust data infrastructure. Today, we will have a glimpse of the Meta internal data stack.

You can find the original article from Meta here.

Data Warehouse

The Meta’s data warehouse is a repository for analytics requirements. It has millions of Hive tables, physically stored using an internal fork of ORC.

Data is spread across namespaces. Namespaces are either physical (geographical) or logical partitioning of the warehouse: tables are grouped into a namespace to efficiently be used in the same queries without moving data around. Thus, data replication is required when tables need to be accessed from two different namespaces.

Data in Meta's systems is kept only as long as needed. Tables in the warehouse usually have a set retention period, such as 90 days, after which older data is either archived or deleted.

Each table is linked to an on-call group, which designates the responsible team and the point of contact for any issues or questions about the data.

Data warehouse ingestion

• Snapshot from operational databases (Meta’s graph database)

• Logs from clients or servers.

• From the Dataswarm pipeline, mainly retrieved by querying other warehouse’s tables.

Data discovery, data catalog

Engineers at Meta developed a web-based tool called iData, which lets users search data assets (tables, dashboards,…) by keyword. The tool also includes lineage tools to help users trace upstream and downstream data assets.

The Query Engine

Meta's data warehouse can be queried, mainly using Presto and Spark. Most of their pipelines and queries are written in either Spark SQL or Presto SQL.

They also use imperative approaches with Spark’s Java, Scala, and Python APIs for complex transformations.

The choice between Presto and Spark depends on the workload. Presto is more efficient for most queries, while Spark handles heavier workloads needing more memory or complex joins.

Presto clusters are sized to handle day-to-day ad-hoc queries, scanning billions of rows in seconds or minutes, depending on complexity.

Scuba: Real-time analytics

Scuba is Meta's real-time data analytics framework. It is widely used by data and software engineers to analyze trends in real-time logging data and by software and production engineers to debug.

Daiquery & Bento: The notebooks

At Meta, data engineers use Daiquery daily. This web-based notebook is a single entry point for querying various data sources, including the warehouse (via Presto or Spark), Scuba, and more. Users can upgrade their Daiquery notebooks to Bento notebooks for more complex query analysis. Bento is Meta’s managed Jupyter Notebook implementation, supporting Python or R code and a variety of visualization libraries in addition to queries.

Unidash: Dashboarding

Unidash is the internal tool data engineers use to create dashboards (you can imagine Apache Superset here). It integrates with Daiquery and many other tools; for example, engineers can write their query in Daiquery, create their graph there, and then export it to a new or existing Unidash dashboard.

Software development

Most engineers at Meta use a customized version of Visual Studio Code as an IDE. It has many custom plugins maintained by internal teams. They also use an fork of Mercurial for source control and a monorepo structure—all data pipelines and most internal tools are in a single repository.

Pipeline Developing

At Meta, engineers mainly develop data pipelines in SQL (for business logic) and wrapped in Python code (for orchestration and scheduling).

Their internal Python library for orchestrating and scheduling pipelines is called Dataswarm, a predecessor to Airflow, and is developed and maintained internally.

Monitoring & operations

Pipeline monitoring is done via a web-based tool called CDM (Central Data Manager), which can be seen as the Dataswarm UI.

This is the entry point to a broader tool:

• Identify failing tasks and find the corresponding logs

• Define and run backfills

• Navigate to upstream dependencies

• Identify upstream blockers

• Notifications

• Set up and monitor data quality checks

Outro

Thank you for reading my note.

Now, let’s check some curated resources I found on the internet last week.

📋 The list

────────

How Discord uses open-source tools for scalable data orchestration & transformation — 11 mins, by Zach Bluhm

To continue delivering seamless service and insightful data analytics, we embraced an ambitious project: to overhaul our data orchestration infrastructure using modern, open-source tools. 

Memory Management in DuckDB — 8 mins, by Mark Raasveldt

In this blog post, we will cover aspects of memory management within DuckDB – and provide examples of where they are utilized.

How Canva collects 25 billion events per day — 10 mins, by Long Nguyen

The architecture of our product analytics event delivery pipeline.

A Brief History of Modern Data Stack — 7 mins, by Ananth Packkildurai

A rise & fall of Modern Data Stack and what comes next?

Booking Deduplication: How Agoda Manages Duplicate Bookings Across Multiple Data Centers (Part 1) — 13 mins, Agoda Engineering Blog

Agoda introduced the booking deduplication feature many years ago to prevent the creation of duplicate bookings.

How data observability fits into the different stages in the data pipeline — 9 mins, by Mikkel Dengsøe

In this article, we’ll look into data observability tools’ role in different parts of the data pipeline and their limitations.

😉 Previously on Dimension

Dimension is my sub-newsletter, where I note things I learn from people smarter than me in data engineering. Here is the latest article

Not subscribed yet? Here you go:

Subscribe now

👋 Hi, my name is Vu Trinh, a data engineer.

I share my lesson and excellent resources to read in this newsletter.

Hope this issue finds you well.

Kafka Tiered Storage at Uber

Intro

Last week, Uber released an article introducing Kafka Tiered Storage; the following texts will cover some insights from that article.

You can find the original article from Uber here.

The Kafka original

At first, Kafka stores the messages in segment files on the broker’s file system. The only way to scale the storage capacity is by adding more machines. Due to Kafka's compute-storage coupling, we must add unused memory and CPUs when we only need more storage.

De-coupling effort

Uber proposed Kafka Tiered Storage (KIP-405) to de-couple the Kafka storage and processing power. The proposal suggests that instead of having one storage system, Kafka will have two storage tiers: local (as the initial design) and remote.

Architecture

A Kafka cluster with tiered storage uses two tiers: local and remote. The local tier consists of the broker's current local storage, while the remote tier extends to storage like HDFS/S3/GCS/Azure. Each tier has its own retention configurations based on size and time. The local tier's retention can be reduced to hours, whereas the remote tier can retain data for days or months. Latency-sensitive applications perform reads from the local tier, leveraging efficient page cache. In contrast, applications needing older data, such as backfill or failure recovery, access the remote tier.

This brings some advantages:

• Allowing Kafka’s storage capacity to scale independently.

• Reducing the local storage burden on brokers, minimizing data transfer during recovery and rebalancing.

• Consumers can access messages on remote storage directly without loading data back to the broker.

• Allowing for longer data retention.

Tiered storage divides a topic partition’s log into two components: local log and remote log. The first consists of local log segments, while the latter consists of remote log segments. The remote log subsystem copies eligible segments from local storage to remote storage. A segment becomes eligible for copying when its end offset is less than the partition's LastStableOffset.

The last stable offset (LSO) is the offset in a user partition such that all lower offsets have been decided and are always present - Source.

Copying to the remote storage

The leader broker for a topic partition will process the copying of the eligible log segments to the remote storage. It copies the log segments from the earliest to the latest in a sequence.

Fetching from remote storage

When a consumer fetch request targets data in remote storage, it is handled by a dedicated thread pool. If the requested offset is available in the broker’s local storage, it is served using the local fetch mechanism. This ensures that local and remote reads do not block each other.

Follower Replication

With tiered storage, followers replicate segments from the leader's local storage and must build auxiliary data (e.g., leader epoch state, producer-ID snapshots) before fetching messages. The follower fetch protocol ensures message consistency and order across replicas, even during cluster changes like broker replacements or failures.

Outro

Speaking of Kafka, I’m currently writing a series of articles about my Kafka learning journey; you can find the first article here. This article is an overview of Kafka and its use on LinkedIn. The following article will be about some of Kafka’s important designs and will be released this Saturday.

📋 The list

────────

Building and scaling Notion’s data lake — 12 mins, by Notion Blog

In the past three years, Notion's data has expanded 10x due to user and content growth, with a doubling rate of 6-12 months. Managing this rapid growth while meeting the ever-increasing data demands of critical product and analytics use cases, especially our recent Notion AI features, meant building and scaling Notion’s data lake. Here’s how we did it.

────────

Canva | Unlocking Efficiency and Performance: Navigating the Spark 3 and EMR 6 Upgrade Journey at Slack — 10 mins, by Slack Engineering Blog

Slack Data Engineering recently underwent data workload migration from AWS EMR 5 (Spark 2/Hive 2 processing engine) to EMR 6 (Spark 3 processing engine). In this blog, we will share our migration journey, challenges, and the performance gains we observed in the process. This blog aims to assist Data Engineers, Data Infrastructure Engineers, and Product Managers who may be considering migrating to EMR 6/Spark 3.

────────

Demystifying Data Flow: ETL and ELT Explained Simply — 8 mins, by

I'll explain ETL and ELT in a clear and understandable way. By the end of this article, you'll have a clear grasp of these essential data processing methods and know precisely when to use each one.

────────

Data pipelines and SCDs — 4 mins, by

The recommended approach for backfilling is not to write ad-hoc SQL but to re-run the pipeline over a specified interval. This is done by designing a pipeline with idempotent transformation code. But what about SCDs?

────────

Data Modelling Using Complex Data Types — 6 mins, by

Complex data types like struct, array, map in modern warehouses are game changer, learn the useful aspects from a Data Engineer.

────────

SQL or Python for Data Transformations? - 9 mins, by Start Data Engineering

By the end of this post, you will understand how the underlying execution engine impacts your pipeline performance. You will have a list of criteria to consider when using Python or SQL for a data processing task. With this checklist, you can use each tool to its benefit.

😉 Previously on Dimension

Dimension is my sub-newsletter, where I note down things I learn from people smarter than me in the data engineering field. Here is the latest article

Not subscribed yet? Here you go:

Subscribe now

👋 Hi, my name is Vu Trinh, a data engineer.

I share my lesson and excellent resources to read in this newsletter.

Hope this issue finds you well.

Paypal - Scaling Kafka

This week, we will see how PayPal manages and operates Kafka to support its data growth. This mini-blog is based on the article Scaling Kafka to Support PayPal’s Data Growth (2023).

Kafka at Paypal

At the time of the article writing, Paypal’s Kafka fleet consists of over 85+ clusters with 1,500 brokers that host over 20,000 topics and close to 2,000 Mirror Makers (used to mirror the data among the clusters). During the 2022 Retail Friday, Kafka traffic volume peaked at about 1.3 trillion messages daily.

PayPal's infrastructure is spread across multiple geographical data centers and security zones. They deploy Kafka clusters across these zones and use Mirror Makers to replicate data between clusters. Client applications connect to the topics on these brokers to publish (write) or subscribe (read) the data in the same or different zone. PayPal internally supports various Kafka’s clients such as Java, Python, Spark, Node,…

Operating Kafka at PayPal had its own set of challenges. With different frameworks and tech stacks, they must invest in building robust tools that help them reduce operational overhead. Each section below will describe the era in which PayPay invested.

Cluster Management

PayPal introduced a few improvements:

• Kafka Config Service: If clients want to interact with the Kafka cluster, they must hardcode the broker IPs in the code. When the brokers are replaced due to upgrades, patching, disk failures, etc., the clients must change the broker IP manually. Kafka Config Service pushes information about a set of bootstrap servers (brokers that host the topics) to all the Kafka clients during initialization. If the broker's details change, the Kafka application only needs to restart so that the config service can push the new configuration for them.

• Kafka Access Control Lists (ACLs): ACLs were onboarded at PayPal to help control access to Kafka clusters via the Simple Authentication and Security Layer (SASL) port. Initially, Kafka allowed connections on plain text ports, and any application could connect to any existing topic.

• PayPal Kafka Libraries: PayPal introduced a set of libraries to ensure security, interoperability and user experience:

  • Resilient Client Library: The resilient client library integrates with the discovery service.

  • Monitoring Library: The monitoring library publishes critical metrics for client applications, which helps monitor the applications’ health.

  • Kafka Security Library: All the applications need SSL authentication to connect to the Kafka clusters. This library pulls the required certificates and tokens to authenticate the application during the startup.

• Kafka QA Platform: The older QA environment has a lot of ad-hoc topics, all hosted on a handful of clusters. PayPal redesigned and introduced a new QA platform that provides a one-to-one mapping between production and QA clusters, following the same security standards as the production setup.

Monitoring and Alerting

PayPal's Kafka platform is tightly integrated with its monitoring and alerting systems. Although Apache Kafka provides many metrics by default, they have optimized a subset for quicker issue identification with minimal overhead. Key metrics from brokers, zookeepers, MirrorMakers, and clients monitor application health and underlying machines, triggering alerts at abnormal thresholds. PayPal also developed a custom Kafka Metrics library to filter metrics.

Enhancements and Automation

PayPal automated CRUD operations for clusters and topics, metadata management:

• Patching security vulnerabilities: All hosts in the Kafka platform must be patched frequently to resolve any security vulnerabilities. Patching usually requires broker restarts, risking under-replicated partitions and data loss. To prevent this, they developed a plugin to check under-replicated partitions before patching, allowing clusters to be patched in parallel while ensuring only one broker is patched at a time.

• Topic Onboarding: Application teams must submit a request via the Onboarding Dashboard to create a new topic. The team reviews the capacity requirements and assigns the topic to an available cluster, determined by the capacity analysis tool integrated into the workflow. A unique token is generated for each new application to authenticate access to the Kafka topic, and ACLs are created based on roles. The application can then successfully connect to the Kafka topic.

  

• MirrorMaker Onboarding:

  

• Repartition Assignment Enhancements: By default, Kafka repartitions all partitions, including those on healthy brokers. PayPal modified this to reassign only under-replicated partitions on affected brokers, avoiding long re-partitioning times. Previously, re-partitioning could make clusters unavailable for days, severely impacting availability.

PayPal’s lessons learned.

• Operating Kafka at a large scale requires tools for regular operations.

• Critical health metrics such as CPU and disk utilization are monitored to ensure high availability and business continuity.

• They introduced ACLs to improve application tracking and security and are on the way to developing automation tools to enhance ACL management.

• Benchmarking cluster performance across various environments (on-premises and cloud) with different configurations has provided insights for operational efficiency.

📋 The list

────────

Introduction to Kafka Tiered Storage at Uber — 9 mins, by Uber Engineering Blog

Uber proposed Kafka Tiered Storage (KIP-405) to avoid tight coupling of storage and processing in a broker. It provides two tiers of storage, called local and remote.

────────

The Rise of the Data Platform Engineer — 6 mins, by Pedram Navid

The next evolution of the role is more akin to a Data Platform Engineer.

────────

Why use Apache Airflow (or any orchestrator)? — 7 mins, by Start Data Engineering

Understanding the needs of complex data pipelines can help you understand the need for a tool like Airflow. This post will cover the three main concepts of running data pipelines: scheduling, orchestration, and Observability.

────────

Modern Good Practices for Python Development — 13 mins, by Stuart Ellis

Python has a long history, and it has evolved over time. This article describes some agreed modern best practices.

────────

Datadog - Timeseries Indexing at Scale — 20 mins, Artem Krylysov and May Lee

This blog post provides an overview of the Datadog time-series databaseseries indexing at scale. We’ll compare the performance and reliability of two generations of indexing services.

────────

No Such Thing As Dirty Data — 3 mins, by Ergest Xheblati

There’s no such thing as “dirty data.” Data is either "fit for purpose" or "unfit for purpose." Data "fit for purpose" requires no changes and can be used as is. Data "unfit for purpose" requires "retrofitting" which will ALWAYS cause problems.

────────

Deploy Python Stream Processing App on Kubernetes — 13 mins, by Jaehyeon Kim

The Flink Kubernetes Operator manages the entire deployment lifecycle of Apache Flink applications, simplifying the deployment and management of Python stream processing applications. This series covers deploying a PyFlink application and a Python Apache Beam pipeline on the Flink Runner on Kubernetes.

😉 Previously on Dimension

Dimension is my sub-newsletter, where I note down things I learn from people smarter than me in the data engineering field. Here is the latest article

Not subscribed yet? Here you go:

Subscribe now

👋 Hi, my name is Vu Trinh, a data engineer.

I share my lesson and excellent resources to read in this newsletter.

Hope this issue finds you well.

Uber’s Batch Data Infrastructure with Google Cloud Platform

Intro

Uber runs one of the largest Hadoop installations in the world. Their Hadoop ecosystem hosts more than 1 exabyte of data across tens of thousands of servers. A few weeks ago, Uber released an article introducing that they’re working with Google Cloud Platform (GCP) to move their batch data analytics and ML training stack to GCP to keep up with Uber's growing needs. Let’s review some key insights from that article.

You can find the original blog post here: Modernizing Uber’s Batch Data Infrastructure with Google Cloud Platform.

Strategy

• Their initial GCP migration strategy is to use cloud object storage for the data lake and migrate the rest of the data stack to cloud IaaS (Infrastructure as a Service). This ensures a quick, minimally disruptive migration.

• They plan to leverage applicable PaaS (Platform as a Service) offerings, e.g., GCP Dataproc or BigQuery

Migration Principles

• Avoid painful migrations for data users: Uber tries to minimize the change for users (e.g., dashboard owners). They will use a cloud storage connector for HDFS compatibility with Google Cloud Storage, leveraging open standards like Apache Parquet, Apache Hudi, Apache Spark, Apache Hadoop YARN, and Kubernetes. This minimizes migration challenges and allows smooth integration of on-prem HDFS services with GCP storage.

• Enhance data access proxies: Uber has developed data access proxies for Presto, Spark, and Hive to hide the underlying compute clusters. Once fully migrated, all queries and jobs submitted to these proxies will be routed to the cloud-based stack.

• Leverage Uber’s container and deployment infrastructure: The batch data stack sits on top of Uber’s infrastructure building blocks, which are built to be agnostic between cloud and on-prem. These platforms allow Uber to expand the batch data ecosystem onto the cloud seamlessly.

• Forecast potential data governance issues from cloud services: Uber will enhance data management services to support only approved data services from the cloud vendor, avoiding future data governance complexities.

Major Workload

• Bucket mapping and cloud resources layout: Formulating the mapping algorithm for migrating HDFS files and directories from the source cluster to cloud objects.

• Security integration: Enable support for all users, groups, and service accounts to continue to be authenticated against the object store data lake and any other cloud PaaS. Also, maintain the same levels of authorized access as on-prem.

• Data replication: HiveSync is a permissions-aware, bi-directional data replication service built at Uber. The goal of this is to extend HiveSync’s capabilities to replicate the on-prem data lake’s data into the cloud-based data lake and corresponding Hive Metastore.

• YARN and Presto clusters: Uber will provision new YARN and Presto clusters on GCP. The existing data access proxies will route traffic to these new cloud-based clusters.

Challenges and Initiatives

Here are some of the significant categories of challenges of this large migration:

• Performance: There are differences in features and performance characteristics between Object Store and HDFS. They will leverage the open-source Hadoop connectors and help evolve them to maximize performance.

• Usage governance: Cloud usage costs can be uncontrollable if not carefully managed. Uber will leverage the cloud’s elasticity to control the costs and partner with the internal capacity engineering team to build more advanced cost tracking. 

• Non-analytics/ML-specific usage of HDFS by applications: Uber teams have used HDFS as a generic file store over the years. They migrate these use cases to other internal blob stores while providing a transparent migration path to avoid disruptions.

• Unknown unknowns: There will be unanticipated challenges. They hope to detect these issues with early end-to-end integrations.

Outro

Uber plans to execute the migration plan over the next several quarters and share its progress through a series of blog posts. You can check the Uber blog here for upcoming posts on their migration journey.

📋 The list

────────

How I built a huge graph database of Netflix's cloud infrastructure — 5 mins, by

In this article, I’ll be going over:

• How a graph database data model would help Netflix manage their cloud security, and what datasets would we need

• How we built a Spring Boot API on top of Postgres to serve most use cases

────────

When and Why to Automate: A Data Engineer's Perspective — 8 mins, by

The goal of this article is to:

• Outline why we automate certain tasks

• Call out some of the reasons not to automate a project

Have you, the reader pause and ask, should I really automate this? With that, let’s dive into why you’d look to automate a task.

How Uber ensures Apache Cassandra®’s tolerance for single-zone failure — 12 mins, by Uber Engineering Blog

This blog shows how we ensured the single-zone failure tolerance for Cassandra and, notably, how we converted the large Cassandra fleet in real-time with zero downtime from non-zone-failure-tolerant to single-zone-failure tolerant.

────────

Debugging Data Pipelines — 5 mins, by

Julien shares an approach to help your data pipeline bug-fixing process be a little less stressful.

────────

Scoping Data Projects: Why Technology Alone Isn’t Enough — 9 mins, by janmeskens

This is the first article in a series about data strategy. In this article, I discuss the challenges in defining data projects. Future articles will delve into the topic of data strategy.

────────

How to learn data engineering — 6 mins, by Christophe Blefari

How to learn data engineering in 2024? This article will help you understand everything related to data engineering.

😉 Previously on Dimension

Dimension is my sub-newsletter, where I note down things I learn from people smarter than me in the data engineering field. Here is the latest article

Not subscribed yet? Here you go:

Subscribe now

👋 Hi, my name is Vu Trinh, a data engineer.

I share my lesson and excellent resources to read in this newsletter.

Hope this issue finds you well.

🚀 Data Infrastructure at Airbnb

Intro

Starting from this issue, I'm introducing a new format to GroupBy. In addition to the usual curated links, I'll be sharing a brief blog/note on my recent learnings and readings in the data/software engineering field. I'll strive to keep it concise, under 7 minutes, to respect your time.

This week is my short note after reading the article about data infrastructure at Airbnb (2016).

Reference: Data Infrastructure at Airbnb (2016)

Airbnb philosophy

The data infrastructure at Airbnb was built up by the following philosophies:

• Open-source: trying to adopt the open-source system; if Airbnb builds something that they find helpful, they will contribute back to the community.

• Prefer standard components and methods: Having intuition about when to build a unique solution and when to adopt an existing solution is essential.

• Scalability: Airbnb had to ensure its infrastructure could scale with the growth of the data.

• Solve real problems by listening to your colleagues: Empathizing with internal data users is essential.

Infrastructure Overview

• Data came from two sources: events from Kafka and MySQL database dumps.

• This source data contains user activity event data and dimensional snapshots.

• There are two Hadoop clusters: Gold and Silver. Critical jobs ran in the Gold environment, and more “relaxed” jobs ran in the Silver one.

• Data in Gold is treated as a single source of truth; data can ONLY be copied from Gold to Silver.

• Despite the isolation, separating into two clusters comes at the cost of data replication and keeping data in sync.

• Treating Hive-managed tables as their central source and sink for data.

• Using Presto for almost all ad hoc queries on Hive-managed tables.

• They built a web-based query engine called Airpal backed by Presto. This is Airbnb's primary interface for users to run SQL.

• They use Airflow for job scheduling.

• Engineers and data scientists working on machine learning will work with Spark.

• Airbnb also leverages Spark for stream processing.

Detailed Look at The Hadoop Cluster

Airbnb made a significant migration for their Hadoop Cluster. Two years before the article writing time, two sets of poorly architected clusters called Pinky and Brain were run on a set of EC2 instances running HDFS with 300 terabytes.

At the time of writing, Airbnb had two separate HDFS clusters (Gold and Silver) with 11 petabytes of data, and they also stored multiple petabytes of data in S3. Here are some problems they have overcome during the migration:

• Running Hadoop on Mesos

  • Lacking visibility into logs and cluster health

  • Hadoop on Mesos could only run MapReduce version 1.

  • Cluster underutilization

  • High operational load and difficulty reasoning about the system

  • Solution: moving away from Mesos.

• Remote reads and writes

  • By storing all the HDFS data in mounted EBS volumes, Airbnb sent large amounts of data over the public Amazon EC2 network for queries. → Agains the Hadoop design of local reads and writes on disks.

  • Moreover, they mistakenly split the data storage across three separate availability zones within a single AWS region. Each zone was designated as its own rack, causing remote reads and writes for the three replicas. → More remote data transfer → Slow performance.

  • Solution: having dedicated instances using local storage and running in a single availability zone.

• Heterogeneous Workload on Homogeneous Machines:

  • There were distinct requirements for the architectural components.

  • Hive/Hadoop/HDFS machines required a lot of storage but didn’t need much RAM or CPU.

  • Presto and Spark required RAM and CPU but didn’t need much storage.

  • Solution: Leveraging the flexibility of EC2 instance types supported by Amazon for each component to save cost and increase resource utilization.

• System Monitoring

  • One major issue was creating custom monitoring and alerting for the cluster. Hadoop, Hive, and HDFS are complex systems with many potential failure points. Anticipating all failure states and setting reasonable alert thresholds felt like reinventing the wheel for Airbnb.

  • Solutions: They signed a support contract with Cloudera to gain from their expertise in architecting and operating these large systems and to reduce the maintenance burden by using the Cloudera Manager tool.

After the migration, they were able to cut costs dramatically and, at the same time, increase awesome performance. Here are a few numbers:

• Disk read/write improved from 70–150MB/sec to 400+ MB/sec

• Read throughput is ~3X better

• Write throughput is ~2X better

• Cost is reduced by 70%

Outro

That’s all for my note this week. I decided to write a note like this to share more about things I’ve learned with you guys.

Through this week's note, I hope to help you look closely into the internal infrastructure of Airbnb. Now it’s time for some cool links I found last week.

📋 The list

────────

Databricks - Open Sourcing Unity Catalog — 12 mins, by Databricks blog

We are excited to announce that we are open-sourcing Unity Catalog, the industry’s first open source catalog for data and AI governance across clouds, data formats, and data platforms.

────────

Build Data Engineering Projects with Free Template — 6 mins, by Start Data Engineering

This post will cover the critical concepts of setting up data infrastructure, development workflow, and a few sample data projects that follow this pattern.

────────

Walmart - Reliably Processing Trillions of Kafka Messages Per Day — 8 mins, by Ravinder Matte

In this article we highlight how Apache Kafka messages are reliably processed at a scale of trillions of messages per day with low cost and elasticity.

────────

How trip.com migrated from Elasticsearch and built a 50PB logging solution with ClickHouse — 20 mins, by Dongyu Lin

This blog article will explain the story of our logging platform, why we initially built it, the technology we used, and finally, our plan for its future on top of ClickHouse leveraging some of the features like SharedMergeTree.

────────

Building an ETL pipeline with Apache Airflow and Visualizing AWS Redshift data using Microsoft Power BI — by Ramses Alexander Coraspe Valdez

The goal of this project is to track the expenses of Uber Rides and Uber Eats through data Engineering processes using technologies such as Apache Airflow, AWS Redshift, and Power BI.

────────

Reducing Data Questions Deluge — 5 mins, by Ergest Xheblati

How properly done self-service analytics can reduce requests on data teams.

────────

Senior Engineer Fatigue — 4 mins, by luminousmen

Senior fatigue is, perhaps paradoxically, a sign of maturity in engineering. It's an indicator that you’re transitioning from doing everything to ensuring that everything that needs to be done gets done in the most effective way.

😉 Previously on Dimension

Dimension is my sub-newsletter, where I note down things I learn from people smarter than me in the data engineering field. Here is the latest article

Not subscribed yet? Here you go:

Subscribe now

👋 Hi, my name is Vu Trinh, a data engineer.

I share my lesson and excellent resources to read in this newsletter.

Hope this issue finds you well.

💠┆How to choose between batch, micro-batch, and streaming when building a data pipeline

• 3 mins, by Zach Wilson

Speed is all the rage in data engineering nowadays. Stakeholders want their data sooner with higher quality and efficiency! All this demand for speed can cause data engineers to make stupid decisions by not asking the proper follow-up questions to decipher the true requirements from the false ones!

💠┆Serverless Jupyter Notebooks at Meta

• 5 mins, by Steve Dini

This blog shows how Meta enable their internal Jupyter Notebook platform, Bento, to run directly on the web browser using a library called Pyodide that sits on top of WebAssembly (Wasm)

💠┆How Stripe’s document databases supported 99.999% uptime with zero-downtime data migrations

• 12 mins, by Jimmy Morzaria, Suraj Narkhede

In this blog post we’ll share an overview of Stripe’s database infrastructure, and discuss the design and application of the Data Movement Platform.

💠┆What I learned after one year of building a Data Platform from scratch

• 9 mins, Jeremy Surget

One year ago, I joined a French start-up called Allowa, which is on a mission to be the marketplace for real estate services. I joined as the first data guy to help structure all their data and ultimately extract value from it. Building a data platform from scratch is an amazing experience and I wanted to share the lessons that I learned along the way.

💠┆Things you wish you didn't need to know about S3

• 8 mins, Daniel Grzelak

This blog gives some weird things you might not know about Amazon S3.

💠┆How we orchestrate 2000+ DBT models in Apache Airflow

• 16 mins, by Alexandre Magno Lima Martins

In this blog, you will find how we used Airflow to orchestrate our DBT Core project, creating an intuitive pipeline that empowered data analysts and even product owners to develop and maintain their own data models.

💠┆Python Essentials for Data Engineers

• 10 mins, by Start Data Engineer

You know Python is important for data engineers. But what does knowing Python mean for data engineering? Python is a programming language that supports a wide range of functions; how would you know if you know it well enough for data engineering?

💠┆What 10 Years at Uber, Meta and Startups Taught Me About Data Analytics

• 9 mins, Torsten Walbaum

Over the last 10 years, I have worked in analytical roles in a number of companies, from a small Fintech startup in Germany to high-growth pre-IPO scale-ups (Rippling) and big tech companies (Uber, Meta). Below, you’ll find ten of my key learnings over the last decade, many of which I’ve found to hold true regardless of company stage, product or business model.

💠┆How To Data Model – Real-Life Examples Of How Companies Model Their Data

• 8 mins, by Seattle Data Guy

How companies’ data model varies widely.

🚀 Previously on Dimension

Dimension is my sub-newsletter, where I note down things I learn from people smarter than me in the data engineering field. Here is the latest article

Not subscribed yet? Here you go:

Subscribe now

👋 Hi, my name is Vu Trinh, a data engineer.

I share my lesson and excellent resources to read in this newsletter.

Hope this issue finds you well.

💠┆Modernizing Uber’s Batch Data Infrastructure with Google Cloud Platform

• 7 mins, by Uber Engineering Blog

• Topics: Data Infrastructure

Uber runs one of the largest Hadoop installations in the world. Our Hadoop ecosystem hosts more than 1 exabyte of data across tens of thousands of servers in each of our two regions. The open-source data ecosystem, including the Hadoop ecosystem discussed in previous engineering blogs, has been the core of our data platform. Today, we are excited to announce that we are working with Google Cloud Platform (GCP) to move our batch data analytics and ML training stack to GCP. 

💠┆Spotify | Data Platform Explained Part II 

• 6 mins, by Spotify Engineering Blog

• Topics: Data Platform

In the second part, Spotify engineer will talk about scalability, the tooling they use and provide, alongside the value each building block brings to a data platform — and finally, their strategy to navigate the complexity of a data ecosystem by building a strong community around it.

💠┆The ABCs of Data Products: An Essential Beginner's Introduction

• 6 mins, by

• Topics: Data Platform

From Concept to Execution: Building Strong Foundations for Data Engineering Success

I’ve just found an awesome newsletter, Data Gibberish, by ; if you’re looking for easy-to-read and valuable content on DataOps, you should visit Yordan’s newsletter.

💠┆Incremental Deprecation

• 2 mins, by

• Topics: Software Engineering

Deprecate incrementally along with incremental releases to save cost, avoid tech debt and more, all in today's article.

💠┆Apache Iceberg - What Is It

• 13 mins, by

• Topics: Storage Formats, Lakehouse

This article consolidates everything I've learned about Iceberg over the past month. It’s a bit long; I hope you will enjoy it!

💠┆Building Cost Efficient Data Pipelines with Python & DuckDB

• 11 mins, by Start Data Engineering

• Topics: Data Processing, HandsOn

In this post, we will discuss an approach that uses the latest advancements in in-memory processing coupled with cheap, powerful hardware to reduce data processing costs significantly!

We will use DuckDB with ephemeral VMs to process data. While DuckDB + VMs are typically cheaper, you will need to be mindful of how you use them. In this post, we will see

💠┆dbt(Data Build Tool) Tutorial

• 10 mins, by Start Data Engineering

• Topics: Data Processing, HandsOn

An excellent and detailed project to help you start with the dbt.

💠┆Building Durable Data Pipelines

• 12 mins, by Mike Shakhomirov

• Topics: Data Pipeline, HandsOn

Data engineering techniques for robust and sustainable ETL

🚀 Previously on Dimension

Dimension is my sub-newsletter, where I note down things I learn from people smarter than me in the data engineering field.

This week is my guest post on Junaid’s newsletter.

Junaid Effendi | Sharing knowledge for Engineers

Everything you need to know about MapReduce

Table of contents The Motivation The Model The MapReduce Implementation Support Features…

Read more

2 years ago · 11 likes · 5 comments · Junaid Effendi and Vu Trinh

Not subscribed yet? Here you go:

Subscribe now

👋 Hi, my name is Vu Trinh, a data engineer.

I share my lesson and excellent resources to read in this newsletter.

Hope this issue find you well.

😊 It’s Vu Trinh here …

My note on learning new things.

For me, Data Engineering requires a lot of skills. From hard to soft ones. Thus, the daily DE job allows me to learn many things.

Besides the daily DE job, as you know, I’ve spent quite a decent amount of time on my newsletter. From the GroupBy to the Dimension, I can learn new things whenever I write.

After being on the journey for a while, here are my few notes about learning new things:

• Before spending time learning something, I ask myself, “Why do I need to learn this?“ and set expectations for what I will learn.

• After ensuring the goal and expectations, I sit down and start learning; the important thing is I try to focus on ONLY ONE thing. For example, if I want to learn SQL window functions, I spend all the learning time on it; I avoid being like “one hour for learning this“or “two hours for learning that. “

• The first step when I start learning is asking, “Why does this exist? It must solve some problems. So what are the problems?“ If you find it hard to imagine, try to answer this, and you will know what I mean: “Why does Apache Airflow exist? What problem does it solve? “

  • I ask these questions every time. From the first time I start learning how to write API to when I start reading papers about OLAP system

• Then, I try to break down the things I will learn into the most basic form. There are specific terms for this approach: First Principle Thinking. There is a video on YouTube where Elon Musk is sharing about this; the video may explain better what I’m saying.

• The above steps will give me the fundamentals of understanding it. After that, I will try to “materialize“ things I have learned: writing a small paragraph, writing a piece of code, deploying this and that. The MOST IMPORTANT thing here is to start simple. I will try a small function or class if I'm learning a new programming language.

  • Trying to do complicated things at first usually prevents me from completing them. This frustrates and demotivates me, and finally, I have to give up. Instead, I will start small at first and improve myself every iteration; then, I will be ready to go with the complicated things.

• Exposing myself. After achieving the learning goal, I will try to expose it. Posting my blog on the writing platform, having seniors reviewing my code,…. I’m trying to get feedback as much as possible.

• I almost forgot this point: It’s okay if I get stuck a little bit. In the past, when I faced obstacles in the process, I felt terrible about myself, “Why I’m so stupid? Why I’m so slow?“. Through time, I realized it’s okay if I get stuck; that’s not a big deal, and the terrible feelings will not solve the problem. Instead, I try to relax myself and enjoy the learning process. Keeping moving is the most critical factor here.

That’s it. These are only my notes, and I do not even strictly follow these things (last week, I tried to jump right into some advanced concept of Rust and felt so lost).

The thing I want to say here is…

… you must have a strategy.

Learning is a lifelong process, and our time is finite, so we need an efficient way to learn.

So, what is your learning strategy?

💠┆Composable data management at Meta

• 9 mins, by Pedro Pedreira, Amit Purohit

• Topics: Data Infrastructure

In recent years, Meta’s data management systems have evolved into a composable architecture that creates interoperability and promotes reusability. We’re sharing how we’ve achieved this, in part, by leveraging Velox, Meta’s open source execution engine, as well as work ahead as we continue to rethink our data management systems. 

💠┆How Uber Accomplishes Job Counting At Scale

• 11 mins, by Uber Engineering Blog

• Topic: Data Processing

Uber operates on a massive scale, facilitating over 2.2 billion trips every quarter. Deriving even simple insights necessitates a scaled solution. In our case, we needed to count the number of jobs someone had participated in while on the Uber platform for arbitrary time windows. This article focuses on the challenges faced and lessons learned as we integrated Apache Pinot™ into our solution.

💠┆What exactly is “tech debt”?

• 5 mins, by Jacob Bennett

• Topics: Software Development

Defining and quantifying technical debt

💠┆What Is the Best Advice You Have Ever Received? - Issue 203

• 15 mins, by Olga Berezovsky

• Topics: Career

What changed or transformed your career? Take some advice from data and analytics leaders.

💠┆How RocksDB works

• 15 mins, by Artem Krylysov

• Topics: OLAP database, deep dive

I spent the past 4 years at Datadog building and running services on top of RocksDB in production. In this post, I'll give a high-level overview of how RocksDB works.

💠┆Pydantic for Experts: Reusing & Importing Validators

• 10 mins, by Yaakov Bressler

• Topics: Python, Coding

Advanced techniques for reusing and importing validation across Python models. Pydantic introduces a new way of thinking about schema enforcement — as policy and as manipulation. In other words, assume a schema can be transformed from a variety of inputs, if all of these fail, then the object is invalid.

💠┆DuckDB vs Polars — Which One Is Faster?

• 10 mins, by Yuki Kakegawa

• Topics: Data Processing

An unofficial benchmark on DuckDB and Polars. Read the article to see how the author carries out the benchmark.

💠┆Virtual Data Environments

• 9 mins, by Julien Hurault

• Topics: Data Infrastructure, Data Architecture, Data Warehouse

The separation of computation and storage offered by many modern data infrastructure tools enables us to design smarter development processes based on virtual data environments.

🚀 Previously on Dimension

Dimension is my sub-newsletter, where I note down things I learn from people smarter than me in the data engineering field. Here is my latest article.

Not subscribed yet? Here you go:

Subscribe now

👋 Hi, my name is Vu Trinh, a data engineer.

I enjoy reading good stuff (related to data and engineering), and this newsletter is my effort on the journey to seek the "good stuff" across the entire Internet.

Hope this issue find you well.

😊 It’s Vu Trinh here …

Sharing

Working with pet projects is a good way to learn data engineering.

There were times I loved finding new pet projects on the internet (like a collector)

I opened the browser and entered something like “data engineering side project.”

I picked the one that excites me the most.

Spending all weekend following the tutorial and finally getting things up and running.

I feel proud of myself. I feel like I have accomplished something.

But two weeks later, I remember nothing. Empty head….

…and you know what? I started looking for different projects to play with.

The loop continues on and on, and I barely learn anything from those projects I’ve done.

I know that many data engineers try to learn by doing as many pet projects as possible, but you will learn nothing just by getting things up and running.

Here are my notes when doing a pet project.

• Set expectations on what you will learn.

• Do things right: don’t just try to get things done, for example

  • Learn the fundamentals and best practices of Docker, Git,…

  • Write and organize code nicely, write tests,…

  • The order of SQL execution,…

  • …

• Try to do the project with your friends. This will give you a good impression of how to work on a project with more than one person.

• When playing with any tools (Spark, Airflow, Git, Docker…), ask, “What problem do these tools try to solve?” and “How are they gonna solve that problem?”

• After doing the project, try to put yourself in the role of a user who needs to consume the data from the data warehouse or the dashboard you’ve just set up, and ask yourself, “Am I satisfied?” “Do I get things I need?”. Because, as a data engineer, most of the time, you will work to serve the internal-data-consumed user

• Apply data modeling to your project.

So, what is your lesson after doing the pet project?

💠┆How We Solve Load Balancing Challenges in Apache Kafka

• 15 mins, Agoda Engineering

• Topics: Kafka, data infrastructure, software engineering

In this blog post, we will introduce the concept of Load Balancing for Kafka-based applications at Agoda. We will also explore the issue of imbalance and discuss strategies for effectively addressing these challenges.

💠┆Why it's hard for data engineers to get promoted after senior engineer

• 5 mins, Zach Wilson

• Topics: Career, Learning

In this article, we’ll be talking about:

• How data engineering lacks as much visibility as other data roles

• Data engineers at some companies are viewed as “less technical”

• Data engineering teams are usually smaller than software engineering teams

• The data engineering individual contributor track “tops out” at data architect, and companies don’t need many of them

💠┆Nimble and Lance: The Parquet Killers

• 10 mins, Chris Riccomini

• Topics: Parquet, Storage Format

In this blog post, the author will dig into Nimble - the new storage format from Meta and Lance from LanceDB- to see if these two storage formats can dethrone the Parquet.

💠┆What we learned after running Airflow on Kubernetes for two years

• 15 mins, Alexandre Magno Lima Martins

• Topics: Infrastructure, Airflow, Kubernetes

With this post, the author wants to share important aspects of our deployment that helped them to achieve a scalable and reliable environment. This article is highly recommended if you are running your Airflow environment of Kubernetes.

💠┆4 Software Design Principles I Learned the Hard Way

• 5 mins, Engineer Codex

• Topics: Software Development

During the design and implementation process, I found that the following list of “rules” kept coming back up over and over in various scenarios. These rules are common enough that I daresay that at least one of them will be useful for a project that any software engineers reading this are currently working on.

💠┆How to save 90% of BigQuery’s storage cost

• 6 mins, Yerachmiel Feltzman

• Topics: GCP Cloud, Operation, Google BigQuery

By changing a simple default configuration, I managed to drop BigQuery storage costs bill by 90% for one of Tikal’s clients.

💠┆How to reduce your Snowflake cost

• 14 mins, Start Data Engineering

• Topics: Snowflake, Operation

In this post we will go over fours strategies that you can follow to lower your Snowflake bill. We will start with some quick wins, analyzing table structure and making changes, consolidating data pipelines, and finally setting up monitoring and alerting to ensure continued cost reduction.

💠┆Implementing Change Data Capture (CDC) with Docker, PostgreSQL, MongoDB, Kafka, and Debezium: A Comprehensive Guide

• 20 mins, Ashkan Golehpour

• Topics: Hands-On, CDC, Docker

A detailed guide on how to set up your own CDC solution with Debezium using docker-compose. You definitely give this a look if you’re looking for a side project to get your hands dirty at the weekend.

💠┆How I build an ETL pipeline with AWS Glue, Lambda, and Terraform

• 15 mins, Lorena Gongang

• Topics: Hands-On, AWS, Terraform

Another great hands-on article to get your weekend busy, if you want to learn more about AWS service and Terraform this one is for you.

🚀 Previously on Dimension

Dimension is my sub-newsletter where I note down things I learn from people smarter than me in the data engineering field. Here are my latest article.

Not subscribed yet? Here you go:

Subscribe now

Get more from Vu Trinh in the Substack app

Available for iOS and Android

Get the app

👋 Hi, my name is Vu Trinh, a data engineer.

I enjoy reading good stuff (related to data and engineering), and this newsletter is my effort on the journey to seek the "good stuff" across the entire Internet.

Hope this issue find you well.

😊 It’s Vu Trinh here …

Hi everyone, base on the survey from previous issue, I will adjust some points on the GroupBy issue:

• There will be more Hands-On articles.

• The number of articles will be kept reasonable. (<=10)

• There will be more detailed descriptions in each article.

• I will start sharing more about my data engineering experience in upcoming issues.

💠┆The Netflix Data Engineering Stack

• Youtube Video, 32 mins, Netflix Data

• Topics: Data Architecture

In this excited talk, we will go through the building blocks of the Netflix Data Engineering stack. Learn more about how batch and streaming data pipelines are built at Netflix.

💠┆Atlassian, Evolve your data platform with a Deployment Capability

• 11 mins, Atlassian Engineering Blog

• Topics: Data Architecture

The data platform team at Atlassian re-built their data platform into an opinionated platform; one that saw the introduction of a novel deployment capability. The new capability takes inspiration from Kubernetes and Micros (Atlassian's internal Platform-as-a-Service) but adjusted to the data domain.

💠┆Building Analytical Apps on the Lakehouse using Apache Hudi, Daft & Streamlit

• 10 mins, Dipankar Mazumdar

• Topics: Lakehouse, Apache Hudi, HandsOn

In this blog, our focus would be on building a data app using data directly from an open lakehouse platform.

💠┆Data Pipeline Design Patterns - #1. Data flow patterns

• 12 mins, By Start Data Engineering

• Topics: Data Pipeline, Design Pattern

This post will cover the typical data pipeline flow design patterns. We will learn about the pros and cons of each design pattern, when to use them, and, more importantly, when not to use them. By the end of this post, you will have an overview of the typical data flow patterns.

💠┆Python One Billion Row Challenge — From 10 Minutes to 4 Seconds

• 12 mins, By Dario Radečić

• Topics: Python, Data Processing, HandsOn

The question of how fast a programming language can aggregate 1 billion rows of data has been gaining traction lately. The author tried to solve this challenge by using pure Python at first. Then, he used external data processing libraries like Pandas, Polars, or event DuckDB to see how it goes—a worth-reading article.

💠┆Coding Data Pipeline Design Patterns in Python

• 10 mins, By Ahmed Sayed

• Topics: Python, Programming, Code Design Pattern, HandsOn

Design patterns are language-independent strategies for solving common design problems. They encapsulate successful design practices and provide a set of proven solutions to design challenges, promoting best practices in software design. They are not mandatory to implement in every project, but using them can make code more flexible, reusable, and maintainable.

💠┆How We Migrated From dbt Cloud and Scaled Our Data Development

• 10 mins, By Gloss Genius Blog

• Topics: Data Architecture

In this blog post, we will share our journey of migrating away from dbt Cloud – exploring the reasons behind this decision, the alternatives we considered and the lessons we learned along the way.

🚀 Previously on Dimension

Dimension is my sub-newsletter where I note down things I learn from people smarter than me in the data engineering field. Here are my latest article.

Not subscribed yet? Here you go:

Subscribe now

Get more from Vu Trinh in the Substack app

Available for iOS and Android

Get the app

👋 Hi, my name is Vu Trinh, a data engineer.

I enjoy reading good stuff (related to data and engineering), and this newsletter is my effort on the journey to seek the "good stuff" across the entire Internet.

Hope this issue find you well.

😊 It’s Vu Trinh here …

🎉🎉 Hi, everyone. I just want to let you know that this newsletter currently has over 1,000 subscribers.

Thank you so much for all of your support.

Reaching a significant milestone like this makes me reflect on my work, especially with the GroupBy newsletter.

The GroupBy newsletter is dedicated to providing you with a curated collection of valuable resources in the field of data engineering, helping you stay updated and informed.

I will put more than 20 links into the GroupBy issue weekly and send them to my audience.

🥶 But I think GroupBy is quite… cold.

I could do better than this.

🔥 From this newsletter, I intend to warm the GroupBy up a little.

So, I need some help.

To prepare better for upcoming issues, I have some short questions below to get your feedback and idea:

If you have any feedback or critiques, please leave a comment or let me know directly via the Substack chat, LinkedIn, or Email.

Once again, thank you so much for your support.

Ah! I almost forgot about the resource list; from now on, the list will be more compact and related.

The number of links will be kept more reasonable to help you focus more.

I will curate the resources more carefully to ensure they are worth your time. 😉

Now, it’s time for this week’s list.

✏️ My curated list

💠┆Hybrid Transactional/Analytical Storage

• 10 mins, by Jack Vanlightly

• Topics: Table Format || Stream Processing || Object Storage

In this article, the author will discuss the Confluent’s vision has for using object storage (with the two recent announcement of Freight Clusters and Tableflow) as an enabler of a new storage paradigm - Hybrid Transactional/Analytical Storage.

💠┆LakeChime: A Data Trigger Service for Modern Data Lakes

• 15 mins, by Linkedin Engineering Blog

• Topics: Data Lake || Table Format || Data Pipeline

In this blog post, we introduce LakeChime, a data trigger service that unifies data trigger semantics not only among modern table formats, but also between modern and traditional table formats such as Hive, bridging the impedance mismatch between traditional partition semantics and modern snapshot semantics.

💠┆From ETL and ELT to Reverse ETL

• 5 mins, by luminousmen

• Topics: Data Pipeline

In recent years, we've witnessed a significant transformation in data management, moving from the traditional ETL (Extract, Transform, Load) framework to the more agile ELT (Extract, Load, Transform) methodology. However, an even newer trend, Reverse ETL, is reshaping our approach to data integration.

💠┆From Predictive to Generative – How Michelangelo Accelerates Uber’s AI Journey

• 20 mins, by Kai Wang, Min Cai, Joseph Wang and Eric Chen from Uber

• Topics: AI || Machine Learning

As Uber’s centralized ML platform, Michelangelo has been instrumental in driving its ML evolution since its introduction in 2016. In this blog post, we will see how Michelangelo empowers many end-to-end high-quality ML applications at Uber.

💠┆Ensuring data reliability and observability in risk systems

• 5 mins, by Yi Ni Ong, Kamesh Chandran and Jia Long Loh from Grab

• Topics: Data Management

As Grab’s business grows, so does the amount of data. Any data discrepancy or missing data surely impact the business operation. They need to quickly detect any data anomalies, which is where data observability comes in.

💠┆The only two log levels you need are INFO and ERROR

• 8 mins, by Nicole Tietz-Sokolskaya

• Topics: Programming || Software Development

It's common for us to log in ways that are unhelpful.

💠┆Database in Kubernetes: Is that a good idea?

• 13 mins, by Vonng

• Topics: Database || Infrastructure

Whether databases should be housed in Kubernetes/Docker remains highly controversial. While Kubernetes (k8s) excels in managing stateless applications, it has fundamental drawbacks with stateful services, especially databases like PostgreSQL and MySQL.

💠┆The Foundation of Data Validation

• 7 mins , by Chengzhi Zhao

• Topics: Data Processing

Discussing the basic principles and methodology of data validation

💠┆Modeling Slowly Changing Dimensions

• 16 mins, by Giorgos Myrianthous

• Topics: Data Modeling

A deep dive into the various SCD types and how they can be implemented in Data Warehouses

💠┆How an empty S3 bucket can make your AWS bill explode

• 4 mins, by Maciej Pocwierz

• Topics: Cloud || Object Storage

Imagine you create an empty, private AWS S3 bucket in a region of your preference. What will your AWS bill be the next morning? Two days later, I checked my AWS billing page, primarily to make sure that what I was doing was well within the free-tier limits; My bill was over $1,300, with the billing console showing nearly 100,000,000 S3 PUT requests executed within just one day!

🚀 Previously on Dimension

Dimension is my sub-newsletter where I note down things I learn from people smarter than me in the data engineering field. Here are my latest article.

Not subscribed yet? Here you go:

Subscribe now

Get more from Vu Trinh in the Substack app

Available for iOS and Android

Get the app

👋 Hi, my name is Vu Trinh, a data engineer.

I enjoy reading good stuff (related to data and engineering), and this newsletter is my effort on the journey to seek the "good stuff" across the entire Internet.

Hope this issue find you well.

🐙 Learning

I love to learn, and I assume you do too.

📖┆Learn one thing at a time

✍ Lawrence Jones

Of the mental models and rules I use in my life, by far the most useful is to learn only one thing at any given time.

🚀 Engineering

I have to believe in a world outside my own mind. — Memento (2000)

📖┆Data Gateway - A Platform for Growing and Protecting the Data Tier

✍ Netflix Technology Blog

The Netflix Online Datastore team has built a platform we call the Data Gateway to enable our datastore engineers to deliver powerful data abstractions which protect Netflix application developers from complex distributed databases and incompatible API changes. In this opening post, we cover the platform as the first part of a series which shows how we use this platform to raise the level of abstraction that application developers use every day to create, access, and maintain their online data.

📖┆Solving RevenueCat's data ingestion challenges into Snowflake

✍ Jesús Sánchez

In this blog, I’ll take you through the intricacies of our data management practices, specifically focusing on the journey of our data from its origins to its final destination in Snowflake. We’ll explore the challenges we faced, the solutions we devised, and the insights we gained through the process of optimizing our data ingestion pipeline.

📖┆The Deconstructed Database

✍ Julien Le Dem

Recently there’s been more talk around the same idea of building composable data systems. Arrow and Iceberg have grown exponentially in popularity and some of the aspirational ideas in my talk in 2018 are now well established. We do live in the future I was hoping for then. In this post, I want to explain in more detail what those components are and, more importantly, the contracts that keep them decoupled and composable.

📖┆The Cloud Storage Triad: Latency, Cost, Durability

✍ Chris Riccomini

I believe that the future of database persistence is object storage—S3, Google Cloud Storage, and so on. New systems like Neon, WarpStream, and Turbopuffer persist data in object storage to offer infinite retention, durability, replication, data warehouse integration, and so on.

📖┆No Memory? No Problem. External Aggregation in DuckDB

✍ Laurens Kuiper

In a nutshell, that’s what this post is about. Since the 0.9.0 release, DuckDB’s hash aggregation can process more unique groups than fit in memory by offloading data to storage. In this post, we’ll explain how this works. If you want to know what hash aggregation is, how hash collisions are resolved, or how DuckDB’s hash table is structured, check out our first blog post on hash aggregation.

📖┆From ZooKeeper to KRaft: How the Kafka migration works

✍ Paolo Patierno

Through this blog post we are going to describe the main differences between using ZooKeeper and KRaft for Kafka to store the cluster metadata and how the migration from the former to the latter works.

📖┆Scaling Kafka by Parallel Processing

✍ Tanuj Kumar

In this blog post, we will delve into the world of parallel consumer strategies for Kafka, exploring various approaches and techniques for achieving parallelism in message processing. We will discuss the benefits and trade-offs of each approach, along with best practices for implementation.

📖┆Designing Kafka Streams Applications

✍ Valérie Servaire + Paul Amar + Damien Fayet+ Sébastien Viale

For the past 4 years, our journey into the heart of Kafka's capabilities has been shaped by two pivotal concepts: Master Topologies and Micro Topologies. These conceptual frameworks have become the backbone of our Kafka Streams application design, offering a comprehensive and granular understanding of our end-to-end communication.

📖┆pg_analytics: Transforming Postgres into a Fast OLAP Database

✍ Ming Ying

We’re excited to introduce pg_analytics, an extension that accelerates the native analytical performance of any Postgres database by 94x. With pg_analytics installed, Postgres is 8x faster than Elasticsearch and nearly ties ClickHouse on analytical benchmarks.

📖┆Data Anywhere with Pipelines, Event Notifications, and Workflows

✍ Matt Silverlock

Data is fundamental to any real-world application: the database storing your user data and inventory, the analytics tracking sales events and/or error rates, the object storage with your web assets and/or the Parquet files driving your data science team, and the vector database enabling semantic search or AI-powered recommendations for your users.

📖┆Investigation of a Cross-regional Network Performance Issue

✍ Netflix Technology Blog

This was a very interesting debugging exercise that covered many layers of Netflix’s stack and infrastructure. While it technically wasn’t the “network” to blame, this time it turned out the culprit was the software components that make up the network (i.e. the TCP implementation in the kernel).

📖┆How to test PySpark code with pytest

✍ Start Data Engineering

Have you worked, or are you working with a code base that “moved fast” but had zero to no tests? Every minor feature request makes you start sweating because looking at your codebase the wrong way makes things explode unpredictably.

📖┆Docker Fundamentals for Data Engineers

✍ Start Data Engineering

Docker can be overwhelming to start with. Most data projects use Docker to set up the data infra locally (and often in production). Setting up data tools locally without Docker is (usually)a nightmare! The official docker documentation, while extremely instructive, does not provide a simple guide covering the basics for setting up data infrastructure.

✏ Data

The one thing that this job has taught me is that truth is stranger than fiction. — Predestination (2014)

📖┆Practical Data Modeling - Chapter 1

✍ Joe Reis

What is data modeling? If you ask a group of people this question, you’ll get as many answers as the number of people you asked. Let’s start by defining data and models and then clarifying what data modeling is and is not.

🤖 AI┆ML┆Data Science

You know, Burke, I don’t know which species is worse. — Ripley, Aliens (1986)

📖┆Linkedin - Musings on building a Generative AI product

✍ Juan Pablo Bottaro

Was it easy to build? What went well and what didn’t? Building on top of generative AI wasn’t all smooth sailing, and we hit a wall in many places. We want to pull back the “engineering” curtain and share what came easy, where we struggled, and what’s coming next.

📖┆What can LLMs never do?

✍ Rohit Krishnan

On goal drift and lower reliability. Or, why can't LLMs play Conway's Game Of Life?

📖┆When Do We Stop Finding New Music? A Statistical Analysis

✍ Daniel Parris

So today, we'll explore how our relationship to music changes with age and the developmental phenomena driving our forever-shifting cultural tastes.

📖┆DragonCrawl: Generative AI for High-Quality Mobile Testing

✍ Uber Engineering Blog

This blog will cover a quick introduction to large language models, deep dive into our architecture, challenges, and results. We will close by touching a little on what is in store for DragonCrawl.

📖┆Airbnb Brandometer: Powering Brand Perception Measurement on Social Media Data with AI

✍ Tiantian Zhang

At Airbnb, we have developed Brandometer, a state-of-the-art natural language understanding (NLU) technique for understanding brand perception based on social media data.

📖┆Shepherd: How Stripe adapted Chronon to scale ML feature development

✍ Benjamin Mears

In 2022 we began a partnership with Airbnb to adapt and implement its platform, Chronon, as the foundation for Shepherd—our next-generation ML feature engineering platform—with a view to open sourcing it. We’ve already used it to build a new production model for fraud detection with over 200 features, and so far the Shepherd-enabled model has outperformed our previous model, blocking tens of millions of dollars of additional fraud per year. While our work building Shepherd was specific to Stripe, we are generalizing the approach by contributing optimizations and new functionality to Chronon that anyone can use.

🔥 Catch up

…Next Saturday night, we're sending you back to the future! — Dr. Emmett Brown, Back to the Future (1985)

📖┆SQL code generation is now available for all BigQuery projects. This feature is available in preview.

📖┆User-defined aggregate functions (UDAFs) that support SQL expressions are in preview. User can create a UDAF with the CREATE AGGREGATE FUNCTION statement.

💠 Previously on Dimension

Dimension is my sub-newsletter where I note down things I learn from people smarter than me in the data engineering field.

Here are the 3 latest articles:

Published on 2024, April 13:

Published on 2024, April 20:

Published on 2024, April 27:

Not subscribed yet? Here you go:

Subscribe now

Get more from Vu Trinh in the Substack app

Available for iOS and Android

Get the app

👋 Hi, my name is Vu Trinh, a data engineer.

I enjoy reading good stuff (related to data and engineering), and this newsletter is my effort on the journey to seek the "good stuff" across the entire Internet.

Hope this issue find you well.

📈 Career

Don't let comfort hold you back.

📖┆Principal Engineer

✍ Alex Ewerlöf

Just as going from Senior Engineer to Staff Engineer required a new skill (soft skills), the Principal Engineer requires a new skill: business skills.

🚀 Engineering

I have to believe in a world outside my own mind. — Memento (2000)

📖┆Scaling to Count Billions

✍ Sangzhuoyang Yu

Since we launched the program 3 years ago, usage of our creator content has doubled every 18 months. Now we pay creators based on billions of content usages each month. This usage data not only includes templates but also images, videos, and so on. Building and maintaining a service to track this data for payment is challenging. This blog post introduces the various architectures we’ve experimented with and the lessons we learned along the way.

📖┆Ensuring Precision and Integrity: A Deep Dive into Uber’s Accounting Data Testing Strategies

✍ Uber Engineering Blog

To maintain these tenets, Financial Accounting Services (FAS) Platform has built robust testing, monitoring, and alerting processes. This encompasses system configuration, business accounting, and external financial report generation.

📖┆Lakehouse Data Platform on Kubernetes

✍ Majid Azimi

Building such a modern, cloud-native lakehouse platform is not only possible but has been made more accessible through the use of open-source technologies. Throughout this article, we will explore the intricacies of a data lakehouse platform, emphasizing how it simplifies the transition to and utilization of data lakehouses. We will also provide a comprehensive guide on constructing an entire data lakehouse ecosystem on Kubernetes, highlighting the steps and strategies involved in leveraging this powerful container-orchestration system to deploy and manage a highly scalable and resilient data platform.

📖┆Building Scalable Real Time Event Processing with Kafka and Flink

✍ Allen Wang

At DoorDash, real time events are an important data source to gain insight into our business but building a system capable of handling billions of real time events is challenging. Events are generated from our services and user devices and need to be processed and transported to different destinations to help us make data-driven decisions on the platform.

📖┆Ten years of improvements in PostgreSQL's optimizer

✍ Ryan Marcus

As a query optimization researcher, I’ve spent the last 10 years of my life playing with, learning from, and building on top of the most sophisticated open source query optimizer out there, PostgreSQL. I recently wondered how much PostgreSQL had improved over the decade since I started working on databases. While changelogs and opinion pieces were plentiful, I couldn’t find any strong empirical comparisons, so I decided to run the join order benchmark (JOB) on PostgreSQL 8 through 16. I recorded the 90th percentile query latency for each database version.

📖┆DuckDB Meets Apache Arrow

✍ Jan Kadlec

You may have heard about DuckDB, Apache Arrow, or both. In this article, I’ll tell you about how we (GoodData) are the first analytics (BI) platform powered by the combination of these technologies. I believe the motivation is evident — performance 🏎️ and developer velocity.

📖┆Powering real-time magic moments through notifications

✍ Whatnot Engineering

Whatnot’s notification system had begun to reach its capacity to expand and meet the organization and our user’s needs. It started as a small class hierarchy in our main python codebase then moved to background tasks powered by RabbitMQ. As time passed, the logic and complexity in the operations grew along with the volume of notifications being sent. In this post, we share our journey building the 3rd iteration of a platform we believe will scale for the long term and share our considerations, hurdles, and lessons learned along the way.

📖┆How we Improved Database Development and CI/CD with Storage Snapshot

✍ Agoda Engineering

Database development is an integral part of software development. However, it demands considerable effort and can present challenges throughout the development and testing stages. At Agoda, we’ve adopted specific methodologies and processes to accelerate and simplify the development workflow. In this blog post, we’ll outline the strategies, insights, and implementation techniques of this process.

📖┆The Design of Everyday APIs

✍ Lynn Root

Implementing an API is an art. It’s the connection between the user and the library itself. How can we optimize that connection to make the experience more pleasing? What makes a user reach for one library over another? What goes into an ergonomic API?

📖┆Memgraph Storage Modes Explained

✍ Katarina Supe

Memgraph is an in-memory graph database that ensures data persistence through ACID compliance by default. While it uses snapshots and write-ahead logs (WAL) for data recovery, in some cases, such additional files and insurance are not necessary. Other databases and analytics tools offer snapshots or WAL, but Memgraph offers both with different storage modes.

📖┆How to think about Internal Data Products as a Data Engineer

✍ Hugo Lu

In this article we’ll dive into how to think about Internal Data Products as a Data Engineer. We’ll answer some common questions about Data Products. I’ll show you how I think about building your first Data Product too.

📖┆Origins of Apache Arrow & Its Role Today

✍ Dipankar Mazumdar

This blog details the origins of Apache Arrow and shows how it fits in today’s constantly changing data landscape. The libraries and tools using Arrow illustrate that while data consumers may not directly consume Arrow, they probably have interacted with it underneath the covers and have taken advantage of numerous tasks.

📖┆Apache Hudi: From Zero To One (10/10)

✍ Shiyan Xu

Throughout the last nine posts, I have explored Hudi concepts pertinent to release 0.14, ideas that are relevant across most of the 0.x versions. For the blog series finale, I aim to cast a glance into the future and delve into the exciting new features in the upcoming 1.0 release. In doing so, this ending post will effectively accomplish the purpose of the series: guiding readers from the foundational beginnings to the groundbreaking future - from zero to one.

📖┆Building a Streaming Lakehouse: Performance Comparison Between Paimon and Hudi

✍ Alibaba Cloud Blog

Apache Paimon and Apache Hudi are widely used data lake storage formats with high write throughput and low-latency query performance. This article compares the performance of Paimon and Hudi on Alibaba Cloud EMR and explores their respective roles in building quasi-real-time data warehouses.

✏ Data

The one thing that this job has taught me is that truth is stranger than fiction. — Predestination (2014)

📖┆Data-as-a-Product and Data-Contract: An evolutionary approach to data maturity

✍ Olivier Wulveryck

In recapping, I have always grappled with one question: where does one begin when seeking to implement the data mesh paradigm? Through the journey of exploring this concept, my most recent and profound insight is: the most strategic starting point lies with the data product.

📖┆Debugging Your Business with Data

✍ Ergest Xheblati

How to diagnose the root cause of a 90% drop in pipeline in 5 minutes. An interview with Abhi Sivasailam and Ankur Chawla of Levers Labs

🤖 AI┆ML┆Data Science

You know, Burke, I don’t know which species is worse. — Ripley, Aliens (1986)

📖┆Notes on how to use LLMs in your product

✍ Will Larson

I’ve been working fairly directly on meaningful applicability of LLMs to existing products for the last year, and wanted to type up some semi-disorganized notes. These notes are in no particular order, with an intended audience of industry folks building products.

📖┆How We Built Slack AI To Be Secure and Private

✍ Slack Engineering Blog

Instead, to inform how we built out Slack AI, we started from first principles. We began with our requirements: upholding our existing security and compliance offerings, as well as our privacy principles like “Customer Data is sacrosanct.” Then, through the specific lens of generative AI, our team created a new set of Slack AI principles to guide us

📖┆Milestones on Our Journey to Standardize Experimentation at The New York Times

✍ Kathy Yang

At The New York Times, most product experiments use our internal experimentation platform, ABRA, which is short for A/B Reporting and Allocation architecture. Here’s a look at an early version. A lot has changed since those days, and we want to share some of the things we’ve learned on our standardization journey. While we use other types of experimentation, such as contextual multi-armed bandits, this article is focused specifically around the ways we’ve improved our A/B testing processes.

🔥 Catch up

…Next Saturday night, we're sending you back to the future! — Dr. Emmett Brown, Back to the Future (1985)

📖┆Announcing General Availability of Ray on Databricks

📖┆Apache Airflow 2.9.0: Dataset and UI Improvements

📖┆BigQuery: The quantified LIKE operator is GA. With this operator, you can check a search value for matches against a list of patterns or an array of patterns, using one of these conditions:

• LIKE ANY: Checks if at least one pattern matches.

• LIKE SOME: Synonym for LIKE ANY.

• LIKE ALL: Checks if every pattern matches.

📖┆BigQuery now supports subqueries in row level access policies. This feature is now in public preview.

📖┆Introducing LLM fine-tuning and evaluation in BigQuery

📖┆Introducing Meta Llama 3: The most capable openly available LLM to date

💠 Previously on Dimension

Dimension is my sub-newsletter where I note down things I learn from people smarter than me in the data engineering field.

Here are the 3 latest articles:

Published on 2024, April 6:

Published on 2024, April 13:

Published on 2024, April 20:

Not subscribed yet? Here you go:

Subscribe now

Get more from Vu Trinh in the Substack app

Available for iOS and Android

Get the app

👋 Hi, my name is Vu Trinh, a data engineer.

I enjoy reading good stuff (related to data and engineering), and this newsletter is my effort on the journey to seek the "good stuff" across the entire Internet.

Hope this issue find you well.

📈 Career

Don't let comfort hold you back.

📖┆The 2024 breaking into data engineering roadmap

✍ Zach Wilson

Knowing where to start and how to get a handle on this requires some guidance. This newsletter is going to unveil all the steps needed to break into data engineering in 2024!

📖┆Certifications & What It Signals?

✍ Koo Ping Shung

I recently had an interesting discussion on certification program, which looking at the current landscape, does feel a strong need for a deeper look into it and where it is needed.

📖┆Mental Health in Software Engineering

✍Vadim Kravcenko

I want to talk about something we don't discuss enough in our field: the mental health of software engineers, especially those of us who've taken on the challenge of leadership.

🚀 Engineering

I have to believe in a world outside my own mind. — Memento (2000)

📖┆Migrating a Trillion Entries of Uber’s Ledger Data from DynamoDB to LedgerStore

✍ Uber Engineering Blog

Last week, we explored LedgerStore (LSG) – Uber’s append-only, ledger-style database. This week, we’ll dive into how we migrated Uber’s business-critical ledger data to LSG. We’ll detail how we moved more than a trillion entries (making up a few petabytes of data) transparently and without causing disruption, and we’ll discuss what we learned during the migration.

📖┆The Pragmatic Approach to Data Movement

✍ Eric Sammer

Data movement is the most stubborn problem in infrastructure.

📖┆Under the Hood of Mixpanel’s Infrastructure

✍ Vijay Jayaram

Mixpanel’s analysis UI is powered by an in-house database called Arb, which is built for ingesting, storing, and querying trillions of events in real-time. This page covers the core aspects of our design, the pain points it eliminates for users, and how it compares to other systems.

📖┆Using Clickhouse to scale an events engine + the article’s comment wall on HackNews

✍ Lago Github Repo

Today, we’re going to explore that decision for a hybrid database stack, and more specifically, why we decided to go with ClickHouse.

📺┆Velox Conference 2024 presentations

📖┆Grab Experiment Decision Engine - a Unified Toolkit for Experimentation

✍ Ruike Zhang + Panos Mavrokonstantis

This article introduces the GrabX Decision Engine, an internal open-source package that offers a comprehensive framework for designing and analysing experiments conducted on online experiment platforms. The package encompasses a wide range of functionalities, including a pre-experiment advisor, a post-experiment analysis toolbox, and other advanced tools. In this article, we explore the motivation behind the development of these functionalities, their integration into the unique ecosystem of Grab’s multi-sided marketplace, and how these solutions strengthen the culture and calibre of experimentation at Grab.

📖┆Slack summary pipeline with dlt, Ibis, and Hamilton

✍ Thierry Jean + DagWorks INC

A lightweight & modern Python ETL stack

📖┆How I became an AST convert

✍ Afzal Jasani

AST stands for abstract syntax tree. The team here at Tobiko Data has written a couple different blogs regarding the topic but I want to convey my journey in understanding ASTs and their purpose. It’s fair to say that my past experiences never warranted me to think this deeply about data structures.

📖┆Introducing Trio | Part III

✍ Eli Hart

Part three on how we built a Compose based architecture with Mavericks in the Airbnb Android app

📖┆A Tale of Two Frameworks: The Domain Graph Service Framework Meets Spring GraphQL

✍ Netflix Technology Blog

Netflix open-sourced the Domain Graph Service (DGS) Framework in early 2021. Since then, the framework has seen widespread adoption across Netflix and many other companies. The DGS Framework provides Java developers with a programming model on top of Spring Boot to create GraphQL services.

📖┆Data Engineering: Architectures & Strategies for Handling Sensitive Data

✍ Hussein Jundi

Strategies and architectures to handle sensitive data efficiently and securely according to an organization’s data maturity level.

📖┆Data Council 2024: The future data stack is composable, and other hot takes

✍ Chase Roberts

I wrote the synopsis below for the Vertex Ventures investment team as a debrief to last week’s Data Council conference. The summary wasn’t originally intended to be published as a blog post, but one of my partners suggested I post it publicly. So, here it is — entirely unedited.

📖┆My fun journey of managing a large table of PostgreSQL

✍ digitake

The consolidator, at first, runs reasonably fast for its size. It takes about a minute to perform an SQL SELECT/GROUP BY/ORDER BY function. One might be surprised that one minute is relatively fast compared to the number of rows in the table. The reason for that is, I do, use an index to keep track of data. I will take you with me through my journey to the data wonderland.

📖┆How we’ve saved 98% in cloud costs by writing our own database

✍ Hivekit Blog

What is the first rule of programming? Maybe something like “do not repeat yourself” or “if it works, don’t touch it”? Or, how about “do not write your own database!”… That’s a good one.

✏ Data

The one thing that this job has taught me is that truth is stranger than fiction. — Predestination (2014)

📖┆Scalable Data Management with Microsoft Fabric and Microsoft Purview

✍ Piethein Strengholt

In this blog post, I aim to demonstrate a practical data transformation using Microsoft Fabric and Microsoft Purview. The target audience includes executives, architects, analysts, and compliance and governance personnel who are interested in creating a comprehensive data platform.

📖┆Microsoft Excel in the Era of Big Data

✍ Benoit Pimpaud

What if we learn about how to build efficient and consistent spreadsheets? More than just getting closer to this engineering stuff, there are a lot of benefits for our daily tasks to strengthen these files with more efficiency, consistency, and reproducibility designs.

📖┆How is the state of analytics engineering?

✍ Dan Poppy

The 2024 State of Analytics Engineering is in. Let's get into it.

🤖 AI┆ML┆Data Science

You know, Burke, I don’t know which species is worse. — Ripley, Aliens (1986)

📖┆Chronon, Airbnb’s ML Feature Platform, Is Now Open Source

✍ Varant Zanoyan

A feature platform that offers observability and management tools, allows ML practitioners to use a variety of data sources, while handling the complexity of data engineering, and provides low latency streaming.

📖┆How machine learning boosted available cash flows for our sellers

✍ Pierre Pessarossi

In this article, we will explain how we created business value by coupling machine learning with a simple sampling technique that bypassed the need to adapt our technical stack.

📖┆4 ways GitHub engineers use GitHub Copilot

✍ Holger Staudacher

GitHub Copilot increases efficiency for our engineers by allowing us to automate repetitive tasks, stay focused, and more.

🔥 Catch up

…Next Saturday night, we're sending you back to the future! — Dr. Emmett Brown, Back to the Future (1985)

📖┆Announcing Delta Lake support for BigQuery

📖┆Apache Kafka for BigQuery

📖┆Gemini in BigQuery and Looker

📖┆BigQuery data canvas

📖┆BigQuery's continuous queries feature, which provides users the ability to run continuously processing SQL statements that can analyze and transform data as new events arrive in BigQuery. This is only available for private preview.

📖┆Next-generation Meta Training and Inference Accelerator

📖┆Apache Superset 4.0

📖┆Introducing Beam YAML: Apache Beam's First No-code SDK

💠 Previously on Dimension

Dimension is my sub-newsletter where I note down things I learn from people smarter than me in the data engineering field.

Here are the 3 latest articles:

Published on 2024, March 30:

Published on 2024, April 6:

Published on 2024, April 13:

Not subscribed yet? Here you go:

Subscribe now

Get more from Vu Trinh in the Substack app

Available for iOS and Android

Get the app

👋 Hi, my name is Vu Trinh, a data engineer.

I enjoy reading good stuff (related to data and engineering), and this newsletter is my effort on the journey to seek the "good stuff" across the entire Internet.

Hope this issue find you well.

🚀 Engineering

I have to believe in a world outside my own mind. — Memento (2000)

📖┆How LedgerStore Supports Trillions of Indexes at Uber

✍ Uber Engineering Blog

This blog covers the significance of LedgerStore indexing and its architecture, which powers trillions of indexes, with a petabyte-scale index storage footprint.

📖┆Composable Data Systems: Lessons from Apache Calcite Success

✍ Vladimir Ozerov

In this blog post, I would like to share our experience with Apache Calcite — a powerful composable toolset for building query optimizers. Apache Calcite achieved tremendous success, powering query optimization in many popular systems, such as Apache Hive and Apache Flink.

📖┆Data Platform Explained

✍ Anastasia Khlebnikova + Carol Cunha

As engineers working at Spotify, we frequently find ourselves explaining our robust data platform to fellow professionals who are contemplating embarking on a similar venture within their organizations. Despite the number of articles, blog posts, and talks one can find online, it can be challenging to digest the information about the building blocks of a data platform, how to start building one, and the tradeoffs to consider for what is good for the business. In this blog post series, we’ll delve into what our data platform entails, its pivotal role at Spotify, and the key factors leading organizations to consider building one.

📖┆Netflix’s Media Landscape Evolution: From 3–2–1 to Cloud Storage Optimization

✍ Netflix Technology Blog

This blog will explore how harnessing user access patterns helped us optimize storage efficiency and cost-effectiveness smartly. Within this exploration, we delve into a cost analysis of lifecycle policies, explicitly examining the cost-effectiveness of various archival and purge strategies tailored to different AWS storage layers.

📖┆The Design Philosophy of Great Tables

✍ Rich Iannone + Michael Chow

Through the exploration of the qualities that make tables shine, the backstory of tables as a display of data, and the issues faced today, it’s clear how we can solve the great table dilemma with Great Tables.

📖┆How we Built a 19 PiB Logging Platform with ClickHouse and Saved Millions

✍ Clickhouse Blog

In the interest of others benefiting from our journey, we provide the details of our own ClickHouse-powered logging solution that contains over 19 PiB uncompressed, or 37 trillion rows, for our AWS regions alone. As a general design philosophy, we aspired to minimize the number of moving parts and ensure the design was as simple and reproducible as possible.

📖┆Small Files Issue: Where Streams and Tables Meet

✍ Hubert Dulay

The announcement that Confluent will now support seamless materialization of Apache Kafka topics as Iceberg Tables (aka Tableflow) has gotten the streaming world and lakehouse worlds rubbing their chins 🤔 and cleaning their monocles 🧐.

📖┆Introducing Trio | Part II

✍ Eli Hart

Part two on how we built a Compose based architecture with Mavericks in the Airbnb Android app

📖┆Reverse Searching Netflix’s Federated Graph

✍ Netflix Technology Blog

As promised in the previous post, we’ll share how we partnered with one of our Studio Engineering teams to build reverse search. Reverse search inverts the standard querying pattern: rather than finding documents that match a query, it finds queries that match a document.

📖┆Iris - Turning observations into actionable insights for enhanced decision making

✍ Huong Vuong + Hai Nam Cao

Our Iris platform bridges the gap between raw data and meaningful insights, serving the needs of data-driven organisations. Specialising in meticulous monitoring and tracking of Spark and Presto jobs, Iris stands as a transformative tool for peak observability and effective decision-making.

📖┆A Sniff Test for Some Query Optimizers

✍ Justin Jaffray

One important part of query planning is performing transformations over queries. Today I want to see how a couple common databases perform on a completely made-up and unrepresentative benchmark.

📖┆Polars vs. pandas: What’s the Difference?

✍ Jodie Burchell

If you’ve been keeping up with the advances in Python dataframes in the past year, you couldn’t help hearing about Polars, the powerful dataframe library designed for working with large datasets.

✏ Data

The one thing that this job has taught me is that truth is stranger than fiction. — Predestination (2014)

📖┆Synchronizing Organizational Decisions

✍ Ergest Xheblati

How metrics trees foster harmonized decisions and drive sustained impact.

📖┆Dissecting What Makes a Data Strategy Fail: Fluff, Challenges, and Objectives

✍ Sven Balnojan PhD

Unfortunately, bad data strategies are everywhere. Sometimes, I find it hard to find examples of good ones! Most data startups start out with a bad strategy. Almost every single internal data initiative I read is bad.

📖┆The Data Analyst Every CEO Wants

✍ Benoit Pimpaud

The following lines list the top 5 things best to learn and focus on for being the analyst that every CEO should have by his side.

🤖 AI┆ML┆Data Science

You know, Burke, I don’t know which species is worse. — Ripley, Aliens (1986)

📖┆Sneak preview of GPT-5!

✍ Gary Marcus

Holy shit! OpenAI just gave me sneak preview early access to GPT-5 (to do some red-teaming) — and it’s incredible!

📖┆When Will the GenAI Bubble Burst?

✍ Gary Marcus

Why and how it could happen in the next 12 months.

📖┆What is retrieval-augmented generation, and what does it do for generative AI?

✍ Nicole Choi

Here’s how retrieval-augmented generation, or RAG, uses a variety of data sources to keep AI models fresh with up-to-date information and organizational knowledge.

📖┆How we built Text-to-SQL at Pinterest

✍ Pinterest Engineering

We took the rise in availability of Large Language Models (LLMs) as an opportunity to explore whether we could assist our data users with this task by developing a Text-to-SQL feature which transforms these analytical questions directly into code.

📖┆Algorithm War Coming! (Part 2)

✍ Koo Ping Shung

The algorithm war will be never-ending. GenAI models will keep on being trained, with each GenAI model successful in certain dimensions. Model that are currently doing better will only be replaced by a later model that is doing better on all coinciding dimensions.

🔥 Catch up

…Next Saturday night, we're sending you back to the future! — Dr. Emmett Brown, Back to the Future (1985)

📖┆BigQuery | Differential privacy is now generally available (GA).

📖┆BigQuery | User now use BigLake to access Delta Lake tables. This feature is available in preview.

📖┆BigQuery | User can now perform model monitoring in BigQuery ML

📖┆BigQuery data clean rooms with analysis rules and enhanced usage metrics are now generally available (GA).

💠 Previously on Dimension

Dimension is my sub-newsletter where I note down things I learn from people smarter than me in the data engineering field.

Here are the 3 latest articles:

Published on 2024, March 23:

Published on 2024, March 30:

Published on 2024, April 6:

Not subscribed yet? Here you go:

Subscribe now

Get more from Vu Trinh in the Substack app

Available for iOS and Android

Get the app

👋 Hi, my name is Vu Trinh, a data engineer.

I enjoy reading good stuff (related to data and engineering), and this newsletter is my effort on the journey to seek the "good stuff" across the entire Internet.

Hope this issue find you well.

🚀 Engineering

I have to believe in a world outside my own mind. — Memento (2000)

📖┆The Sisyphean struggle and the new era of data infrastructure

✍ Jack Vanlightly

For a while now, I’ve been spending a lot of time thinking about technology trends in the data infrastructure space (as a researcher at Confluent). These trends are making some previously difficult things easy and therefore, commodity. I would go as far as to say that we are witnessing a kind of phase change, a regime shift, at least in the cloud. Almost inevitably, the quote above brought me back to this topic as it revolves around the subject of commodity and competition.

📖┆Ce n'est pas un Kafka: Kafka is a Protocol

✍ Chris Riccomini

Apache Kafka is an aging open source project. It's time to accept that Kafka's protocol is what matters.

📖┆The Data Streaming Landscape 2024

✍ Kai Waehner

This blog post explores the data streaming landscape of 2024 to summarize existing solutions and market trends. The end of the article gives an outlook to potential new entrants in 2025.

📖┆GrafLI - An out-of-the-box Azure monitoring and visualization platform

✍ Prateek Singh

Recognizing these challenges, the Productivity Engineering team at LinkedIn crafted GrafLI, a cloud-native data visualization tool designed to transform the visualization of Azure and on-premises services. In this post, we delve into the intricacies of GrafLI and how it enhances the developer experience and increases engineering velocity.

📖┆Navigating the Netflix Data Deluge: The Imperative of Effective Data Management

✍ Netflix Technology Blog

In this article, we, the Media Infrastructure Platform team, outline the development of a Garbage Collector, our solution for effectively managing production data.

📖┆Columnar DB File Reader V2: A Complete Rewrite

✍ John Mikhail

One of the main pillars of Mixpanel is our proprietary columnar store database, ARB, which we specifically designed to meet the needs of our customers. In this blog post, we delve into a comprehensive rewrite of the event reader code responsible for parsing the columnar files. The primary objective is to significantly enhance query performance, particularly for those with selective filters.

📖┆Anatomy of a lakeFS Repository: Practical Example of Git for Data

✍ Oz Katz

To help you understand the value of Git for data, here’s an overview of the tools that are part of every lakeFS repository. They’re bound to help with your data strategy and ensure that your organization meets its compliance, quality, and safety requirements.

📖┆Introducing Trio | Part I

✍ Eli Hart

A three-part series on how we built a Compose-based architecture with Mavericks in the Airbnb Android app.

📖┆Using GitHub Copilot in your IDE: Tips, tricks and best practices

✍ Kedasha Kerr

In this blog post, I’ll share some of the daily things I do to get the most out of GitHub Copilot. I hope these tips will help you become a more efficient and productive user of the AI assistant.

📖┆Building a Modern Data Service Layer with Apache Arrow

✍ Jan Soubusta

Our journey, led by Lubomir (lupko) Slivka, aims to revolutionize GoodData’s analytics offerings, transforming our traditional BI platform into a robust Analytics Lake. This transformation was motivated by the need to modernize our stack, taking full advantage of open-source technologies and modern architectural principles to better integrate with cloud platforms.

📖┆Cost Optimization Strategies for scalable Data Lakehouse

✍ Suresh Hasundi

In this blog, we will be discussing about the cost related challenges we had faced when our Data Lakehouse scaled and how we have overcome such costs by optimizing the process.

📖┆Why Do Python Lists Multiply Oddly? Exploring the CPython Source Code

✍ Abhinav Upadhyay

We will start by a high level answer by just doing some inspection in the REPL, then we will go one level deeper and see the details of the list implementation in CPython to see why that happens, and finally we will go another level down to see how CPython invokes this behavior.

📖┆PySpark in 2023: A Year in Review

✍ Databricks Blog

With the releases of Apache Spark 3.4 and 3.5 in 2023, we focused heavily on improving PySpark performance, flexibility, and ease of use. This blog post walks you through the key improvements.

📖┆Setting Up Kafka Multi-Tenancy

✍ Yunji Zhong + Amit Gud + Carlos Herrera

In such a multi-tenant architecture, the isolation is implemented at the infrastructure layer. We will delve here into how we set up multi-tenancy with a messaging queue system based on Kafka.

📖┆Data Platforms : Good Architect — Bad Architect

✍ Nilay Shah

Data Engineering is a dynamic field that requires a deep understanding of both technical skills and overarching principles. This transition is often marked by a shift from focusing on immediate technical challenges to embracing a broader perspective on data architecture. If you are Data Architect and want to learn some basic practices — This Article is for you!

📖┆Many Articles Tell You Python Tricks, But Few Tell You Why

✍ Christopher Tao

In my opinion, it is more important to understand the reason why these tricks are there, so we can understand when to use and when not to use them. In this article, I’ll pick up three of them and provide a detailed explanation of the mechanisms under the hood.

🤖 AI┆ML┆Data Science

You know, Burke, I don’t know which species is worse. — Ripley, Aliens (1986)

📖┆Scaling AI/ML Infrastructure at Uber

✍ Uber Engineering Blog

As the complexity and scale of AI/ML models continue to surge, there’s a growing demand for highly efficient infrastructure to support these models effectively. Over the past few years, we’ve strategically implemented a range of infrastructure solutions, both CPU- and GPU-centric, to scale our systems dynamically and cater to the evolving landscape of ML use cases. This evolution has involved tailored hardware SKUs, software library enhancements, integration of diverse distributed training frameworks, and continual refinements to our end-to-end Michaelangelo platform.

📖┆The race between positive and negative applications of Generative AI is on – and not looking pretty

✍ Gary Marcus

OpenAI’s VP of Global Affairs Anna Makanju is exactly right - the race is on. I have some concerns, partly about the way that race is going, partly about (in)justice in who is likely to pay for the costs.

🔥 Catch up

…Next Saturday night, we're sending you back to the future! — Dr. Emmett Brown, Back to the Future (1985)

📖┆Databricks: Announcing the State Reader API: The New "Statestore" Data Source

📖┆Apache Flink Kubernetes Operator 1.8.0 Release Announcement

📖┆BigQuery Query optimization using search indexes is now (GA) applied to comparisons of string literals and indexed data, including the equal (=), IN, and LIKE operators and the STARTS_WITH function.

📖┆The Help me code tool lets user use natural language to generate a SQL query that can then be run in BigQuery.

💠 Previously on Dimension

Dimension is my sub-newsletter where I note down things I learn from people smarter than me in the data engineering field.

Here are the 3 latest articles:

Published on 2024, March 16:

Published on 2024, March 23:

Published on 2024, March 30:

Not subscribed yet? Here you go:

Subscribe now

Get more from Vu Trinh in the Substack app

Available for iOS and Android

Get the app

👋 Hi, my name is Vu Trinh, a data engineer.

I enjoy reading good stuff (related to data and engineering), and this newsletter is my effort on the journey to seek the "good stuff" across the entire Internet.

Hope this issue find you well.

📈 Career

Don't let comfort hold you back.

📖┆How to Start Google

✍ Paul Graham

Starting your own company can mean anything from starting a barber shop to starting Google. I'm here to talk about one extreme end of that continuum. I'm going to tell you how to start Google.

📖┆How to convert your yearly review to a promotion in big tech

✍ Zach Wilson

The dreaded review cycle usually comes once or twice a year at companies. For most this process is annoying or anxiety-provoking. It is also one of the most critical moments for advancing your career with interviewing well being the only one thing that’s more critical!

📖┆Leadership requires taking some risk

✍ Will Larson

At a recent offsite with Carta’s Navigators, we landed on an interesting topic: leadership roles sometimes mean that making progress on a professional initiative requires taking some personal risk.

🚀 Engineering

I have to believe in a world outside my own mind. — Memento (2000)

📖┆Unexplanations: sql is syntactic sugar for relational algebra

✍ Jamie Brandon

This idea is particularly sticky because it was more or less true 50 years ago, and it's a passable mental model to use when learning sql. But it's an inadequate mental model for building new sql frontends, designing new query languages, or writing tools likes ORMs that abstract over sql.

📖┆Tableflow: The Stream/Table, Kafka/Iceberg Duality

✍ Jack Vanlightly

Confluent just announced Tableflow, the seamless materialization of Apache Kafka topics as Apache Iceberg tables. This announcement has to be the most impactful announcement I’ve witnessed while at Confluent. This post is about why Iceberg tables aren’t just another destination to sync data to; they fundamentally change the world of streaming. It’s also about the macro trends that have led us to this point and why Iceberg (and the other table formats) are so important to the future of streaming.

📖┆Kafka tiered storage deep dive

✍ Federico Valeri + Luke Chen

Tiered storage is a new early access feature available as of Apache Kafka 3.6.0 that allows you to scale compute and storage resources independently, provides better client isolation, and allows faster maintenance of your Kafka cluster. Let's dive into this new feature to see the motivations, design, and implementation details. In this post, we will focus on Tiered storage implementation, so it is assumed a good understanding of the Kafka architecture and main components

📖┆Consumer-Driven Contract Testing (CDCT)

✍ Mihriban Kumarci

Consumer-Driven Contract (CDC) Testing is gaining prominence in microservices architecture. It offers an efficient way to ensure that services meet their contracts without exhaustive end-to-end tests..

📖┆TimescaleDB and the Quest for the Ultimate Time Series Database: Growth, Challenges, and Strategic Moves

✍ Sven Balnojan PhD

The one key question I want to dive into today is…. Is Timescale able to handle the coming torrent? Will they be swept away or catch the current and grow absurdly quickly?

📖┆Python @ Picnic

✍ Sven Arends

From Machine Learning to black box testing of Java services, Picnic uses Python throughout its tech stack. Learn why Picnic chose Python and how we leverage its flexibility at scale.

📖┆Every dunder method in Python

✍ Trey Hunner

Python includes tons of dunder methods ("double underscore" methods) which allow us to deeply customize how our custom classes interact with Python's many features. What dunder methods could you add to your class to make it friendly for other Python programmers who use it?

📖┆Real-time Fraud Detection with Yoda and ClickHouse

✍ Nick Shieh

Our Fraud Platform team developed Yoda, a decision platform service, to detect such fraudulent activities quickly and take appropriate measures.To enable fraud decisions in fractions of a second, Yoda uses ClickHouse as its primary real-time datastore. ClickHouse is a fast and highly performant analytical database, widely used across Instacart to power other use-cases such as critical retailer and ads dashboards, calculating results for A/B testing, and machine learning signals.

✏ Data

The one thing that this job has taught me is that truth is stranger than fiction. — Predestination (2014)

📖┆dbt’s Model Contracts for Dummies

✍ Faith Lierheimer

So yeah. The Data Contracts Content Explosion on the Data Internet was fucking exhausting. Unfortunately, it was on to an important idea.

📖┆The Problem with Data Governance

✍ Eric Sandosham, Ph.D.

Over the years, Data Governance has lost much of its lustre having been unable to show much business impact. But what exactly is Data Governance? Has it evolved beyond data quality and data access? How does it square up with the emergence of AI solutioning? And so I dedicate my 29th weekly article to discussing the topic of whether Data Governance as a practice still makes sense.

🤖 AI┆ML┆Data Science

You know, Burke, I don’t know which species is worse. — Ripley, Aliens (1986)

📖┆Let's talk about AI...

✍ Tristan Handy

The integration of structured data and AI will be driven by metadata. And dbt’s biggest role to play in the AI revolution will be as the source of truth for that metadata.

📖┆Logarithm: A logging engine for AI training workflows and services

✍ Partha Kanuparthy

Logarithm is a hosted, serverless, multitenant service, used only internally at Meta, that consumes and indexes these logs and provides an interactive query interface to retrieve and view logs. In this post, we present the design behind Logarithm, and show how it powers AI training debugging use cases.

📖┆Model Excellence Scores: A Framework for Enhancing the Quality of Machine Learning Systems at Scale

✍ Uber Engineering Blog

By integrating the Service Level Agreement (SLA) concept, we aim to establish a standard for measuring and ensuring ML model quality.

📖┆Optimizing RTC bandwidth estimation with machine learning

✍ Santhosh Sunderrajan

We’ve adopted a machine learning (ML)-based approach that allows us to solve networking problems holistically across cross-layers such as BWE, network resiliency, and transport. We’re sharing our experiment results from this approach, some of the challenges we encountered during execution, and learnings for new adopters.

📖┆Sequential A/B Testing Keeps the World Streaming Netflix Part 2: Counting Processes

✍ Netflix Technology Blog

Netflix monitors a large suite of metrics, many of which can be classified as counts. These include metrics such as the number of logins, errors, successful play starts, and even the number of customer call center contacts. In this second installment, we describe our sequential methodology for testing count metrics, outlined in the NeurIPS paper Anytime Valid Inference for Multinomial Count Data.

📖┆Bye Bye Bye...: Evolution of repeated token attacks on ChatGPT models

✍ Mark Breitenbach + Adrian Wood

This blog will discuss the steps taken to execute the repeated token attack on ChatGPT models at various points from October 2023 through March 2024—before, during, and after OpenAI’s filtering mitigation was deployed.

🔥 Catch up

…Next Saturday night, we're sending you back to the future! — Dr. Emmett Brown, Back to the Future (1985)

📖┆dbt 1.8.0

📖┆Apache Flink 1.19

📖┆Apache Kafka 3.7

📖┆BigQuery | Incremental materialized views now support LEFT OUTER JOIN and UNION ALL.

📖┆Microsoft introduces Garnet, a new remote cache-store

📖┆X AI open-sources Grok

📖┆Confluent introduces Tableflow

💠 Previously on Dimension

Dimension is my sub-newsletter where I note down things I learn from people smarter than me in the data engineering field.

Here are the 3 latest articles:

Published on 2024, March 9:

Published on 2024, March 16:

Published on 2024, March 23:

Not subscribed yet? Here you go:

Subscribe now

Get more from Vu Trinh in the Substack app

Available for iOS and Android

Get the app

👋 Hi, my name is Vu Trinh, a data engineer.

I enjoy reading good stuff (related to data and engineering), and this newsletter is my effort on the journey to seek the "good stuff" across the entire Internet.

Hope this issue find you well.

📈 Career

Don't let comfort hold you back.

📖┆The demise of coding is greatly exaggerated

✍ Murat Demirbas

I like to mention that a career in computer science and software technology (practicing coding) gives you vital and generally applicable skills: hacking, debugging, abstract thinking, quick learning/adaptation, and organizational skills.

📖┆40 years of programming

✍ Lars Wirzenius

In April, 1984, my father bought a computer for his home office, a Luxor ABC-802, with a Z80 CPU, 64 kilobytes of RAM, a yellow-on-black screen with 80 by 25 text mode, or about 160 by 75 pixels in graphics mode, and two floppy drives. It had BASIC in its ROM, and came with absolutely no games. If I wanted to play with it, I had to learn how to program, and write my own games. I learned BASIC, and over the next few years would learn Pascal, C, and more. I had found my passion. I was 14 years old and I knew what I wanted to do when I grew up.

📖┆Measuring Developer Productivity via Humans

✍ Abi Noda + Tim Cochran

Measuring developer productivity is a difficult challenge. Conventional metrics focused on development cycle time and throughput are limited, and there aren't obvious answers for where else to turn. Qualitative metrics offer a powerful way to measure and understand developer productivity using data derived from developers themselves.

🚀 Engineering

I have to believe in a world outside my own mind. — Memento (2000)

📖┆Balancing HDFS DataNodes in the Uber DataLake

✍ Uber Engineering Blog

Uber has one of the largest HDFS deployments in the world, with exabytes of data across tens of clusters. It is important, but also challenging, to keep scaling our data infrastructure with the balance between efficiency, service reliability, and high performance.

📖┆We built a new SQL Engine on Arrow and DataFusion

✍ Micah Wylde

Arroyo 0.10 has an entirely new SQL engine built with Apache Arrow and DataFusion. It's much faster, smaller, and easier to run. Read on for why and how we're making this change.

📖┆Differential storage: a key building block for a DuckDB-based data warehouse

✍ Joseph Hwang

Today we’d like to talk about Differential Storage, a key infrastructure-level enabler of new capabilities and stronger semantics for MotherDuck users. Thanks to Differential Storage, features like efficient data sharing and zero-copy clone are now available in MotherDuck. Moreover, Differential Storage unlocks other features, like snapshots, branching and time travel which we’ll release in the coming months.

📖┆Improving Efficiency Of Goku Time Series Database at Pinterest (Part 2)

✍ Pinterest Engineering

This 2nd blog post focuses on how Goku time series queries were improved. We will provide a brief overview of Goku’s time series data model, query model, and architecture. We will follow up with the improvement features we added including rollup, pre-aggregation, and pagination.

📖┆Scaling Models And Multi-Tenant Data Systems - ASDS Chapter 6

✍ Jack Vanlightly

What is scaling in large-scale multi-tenant data systems, and how does that compare to single-tenant data systems? How does per-tenant scaling relate to system-wide scaling? How do scale-to-zero and cold starts come into play? Answering these questions is chapter 6 of The Architecture of Serverless Data Systems.

📖┆How Figma’s databases team lived to tell the scale

✍ Sammy Steele

Figma’s database stack has grown almost 100x since 2020. This is a good problem to have because it means our business is expanding, but it also poses some tricky technical challenges

📖┆Data Engineering Best Practices - #2. Metadata & Logging

✍ Joseph M.

Dealing with breaking pipelines, debugging why they failed, and putting up a fix are everyday tasks for a data engineer.

📖┆S3 is files, but not a filesystem

✍ Cal Paterson

"Deep" modules, mismatched interfaces - and why SAP is so painful

📖┆Building data abstractions with streaming at Yelp

✍ Hakampreet Singh Pandher

This blog post covers how we leverage Yelp’s extensive streaming infrastructure to build robust data abstractions for our offline and streaming data consumers. We will use Yelp’s Business Properties ecosystem (explained in the upcoming sections) as an example.

📖┆Airflow & Kestra: a Simple Benchmark

✍ Benoit Pimpaud

This post compares Airflow and Kestra, focusing on installation, configuration, pipeline syntax, and performance.

📖┆Postgres Aurora DB major version upgrade with minimal downtime

✍ Jay Patel

Our payment platform team had the unique challenge to upgrade our Aurora Postgres DB from v10 to v13. This DB was responsible for storing transactions within Lyft and contains ~400 tables (with partitions) and ~30TB of data. Upgrading the database in-place would have resulted in ~30 mins of downtime. Such significant downtime is untenable — it would cause cascading failures across multiple downstream services, requiring a large amount of engineering effort to remediate.

📖┆Apache Druid’s Architecture – How Druid Processes Data In Real Time At Scale

✍ Ben Rogojan

Apache Druid has several unique features that allow it to be used as a real-time OLAP. Everything from its various nodes and processes that each have unique functionality that let it scale to the fact that the data is indexed to be pulled quickly and efficiently.

📖┆How to save millions by optimizing data pipeline shuffling

✍ Zach Wilson

In this article we will be going over: - Why does shuffle happen and what SQL keywords trigger shuffle and which do not? - Some techniques you can use to minimize shuffle especially in Apache Spark

📖┆A Look Back at Key Trends in Data Infrastructure in 2023 by Four Industry Founders

✍ RisingWave Labs

The discussion with the four founders of data infrastructure startups focused on key trends in the industry for 2023.

📖┆Unlocking Kafka's Potential: Tackling Tail Latency with eBPF

✍ Maciej Mościcki + Piotr Rżysko

At Allegro, we use Kafka as a backbone for asynchronous communication between microservices. With up to 300k messages published and 1M messages consumed every second, it is a key part of our infrastructure. A few months ago, in our main Kafka cluster, we noticed the following discrepancy: while median response times for produce requests were in single-digit milliseconds, the tail latency was much worse. Namely, the p99 latency was up to 1 second, and the p999 latency was up to 3 seconds. This was unacceptable for a new project that we were about to start, so we decided to look into this issue. In this blog post, we would like to describe our journey — how we used Kafka protocol sniffing and eBPF to identify and remove the performance bottleneck.

✏ Data

The one thing that this job has taught me is that truth is stranger than fiction. — Predestination (2014)

📖┆Scalable Automated Config-Driven Data Validation with ValiData

✍ Bharadwaj Jayaraman

ValiData is a scalable automated config-driven data validation tool extensively used in LinkedIn that compares metric values of test datasets against production or source-of-truth datasets and highlights differences in metric values across dimensions.

🤖 AI┆ML┆Data Science

You know, Burke, I don’t know which species is worse. — Ripley, Aliens (1986)

📖┆How Meta tests products with strong network effects

✍ Analytics at Meta

I’m a member of a team that’s been applying cluster experimentation to products with strong network effects, such as chat and calling, since 2018. Today, I’d like to give an overview of the challenges we face in these highly-interactive domains, and how one solution — cluster experiments — has become a go-to method for addressing these challenges.

📖┆Best practices for building LLMs

✍ Nitzan Gado + Oren Dar

Intuit shares what they've learned building multiple LLMs for their generative AI operating system.

📖┆Improving ETAs with Multi-Task Models, Deep Learning, and Probabilistic Forecasts

✍ Doordash Engineering Blog

The DoorDash ETA team is committed to providing an accurate and reliable estimated time of arrival (ETA) as a cornerstone DoorDash consumer experience. We want to ensure that every customer can trust our ETAs, ensuring a high-quality experience in which their food arrives on time every time.

📖┆Building Meta’s GenAI Infrastructure

✍ Kevin Lee + Adi Gangidi + Mathew Oldham

Marking a major investment in Meta’s AI future, we are announcing two 24k GPU clusters. We are sharing details on the hardware, network, storage, design, performance, and software that help us extract high throughput and reliability for various AI workloads. We use this cluster design for Llama 3 training.

🔥 Catch up

…Next Saturday night, we're sending you back to the future! — Dr. Emmett Brown, Back to the Future (1985)

📖┆LinkedIn Open Sources OpenHouse: A Control Plane for Managing Tables in a Data Lakehouse

📖┆OpenTable now changes name to Apache XTable

📖┆Announcing Apache Arrow DataFusion Comet

💠 Previously on Dimension

Dimension is my sub-newsletter where I note down things I learn from people smarter than me in the data engineering field.

Here are the 3 latest articles:

Published on 2024, March 2:

Published on 2024, March 9:

Published on 2024, March 16:

Not subscribed yet? Here you go:

Subscribe now

Get more from Vu Trinh in the Substack app

Available for iOS and Android

Get the app

👋 Hi, my name is Vu Trinh, a data engineer.

I enjoy reading good stuff (related to data and engineering), and this newsletter is my effort on the journey to seek the "good stuff" across the entire Internet.

Hope this issue find you well.

Referral Program

I’m launching a referral program to grow the community by giving you guys valuable gifts whenever you reach a referral milestone. The condition is simple: you refer friends to subscribe to my newsletter, and you will receive a gift based on the number of friends you refer. Here are the reward milestones:

Now, let’s refer friends and claim exciting rewards ;)

Refer a friend

📈 Career

Don't let comfort hold you back.

📖┆Hard and soft skills for developers coding in the age of AI

✍ Sara Verdi

While AI revolutionizes software development, it still relies on developers to pilot its use. In this blog, we’ll cover the skills that developers need to have for navigating this new AI-powered coding frontier.

📖┆The Best Piece of Software Engineering Advice - Confessions of a Data Guy

✍ Daniel Beach

You don’t need to be the smartest person in the room. In fact, you shouldn’t be.

📖┆Why the 100x analyst doesn’t exist

✍ Mikkel Dengsøe

Much has been written about the 10x – heck, 100x – engineer. The mystical creature that ships features in hours instead of months and perseveres where others back down. Corny as it may sound, I think it’s a pretty good representation of how the world works. But does the 10x analyst exist? I believe so. How about the 100x analyst? I’m not sure.

📖┆How to Build a Modern Data Team? Seven tips for success

✍ Martin Chesbrough

I spend my time with clients of Everest Engineering to help them build data platforms, data products and data teams. Learning from them let me share the 7 things that I think are important.

🚀 Engineering

I have to believe in a world outside my own mind. — Memento (2000)

📖┆Load Balancing: Handling Heterogeneous Hardware

✍ Uber Engineering Blog

This blog post describes Uber’s journey towards utilizing hardware efficiently via better load balancing. The work described here lasted over a year, involved engineers across multiple teams, and delivered significant efficiency savings. The article covers the technical solutions and our discovery process to get to them–in many ways, the journey was harder than the destination.

📖┆PERF IS NOT ENOUGH

✍ Jordan Tigani

Performance in general, and general-purpose benchmarking in particular, is a poor way to choose a database. You’re better off making decisions based on ease of use, ecosystem, velocity of updates, or how well it integrates with your workflow. At best, performance is a point-in-time view of the time it will take to complete certain tasks; at worst, however, it leads you to optimize for the wrong things.

📖┆What if we rotate pairs every day?

✍ Gabriel Robaina + Kieran Murphy

We developed a lightweight methodology to help teams reflect on the benefits and challenges of pairing and how to solve them. Initial fears were overcome and teams discovered the benefits of frequently rotating pairs. We learned that pair swapping frequently greatly enhances the benefits of pairing. Here we share the methodology we developed, our observations, and some common fears and insight shared by the participating team members.

📖┆Keeping repository maintainer information accurate

✍ Zack Koppert

Discover how keeping repository maintainer information accurate through CODEOWNERS files and automating maintenance with tools like cleanowners fosters efficient collaboration and sustainable software projects.

📖┆Simplify PySpark testing with DataFrame equality functions

✍ Haejoon Lee + Allison Wang + Amanda Liu

Introducing PySpark DataFrame equality test functions: a new set of test functions in Apache Spark. Discover how easy it is to validate data transformations with the new functions through hands-on examples.

📖┆How GitHub uses merge queue to ship hundreds of changes every day

✍ Will Smythe + Lawrence Gripper

Here's how merge queue transformed the way GitHub deploys changes to production at scale, so you can do the same for your organization.

📖┆Python Upgrade Playbook

✍ Aneesh Agrawal

In this post, we’ll cover how Lyft upgrades Python at scale — 1500+ repos spanning 150+ teams — and the latest iteration of the tools and strategy we’ve built to optimize both the overall time to upgrade and the work required from our engineers. We’ve successfully used (and evolved) this playbook over multiple upgrades, from Python 2 to Python 3.10 and hope you find it useful!

📖┆Building Scalable, Real-Time Chat to Improve Customer Experience

✍ Uber Engineering Blog

With millions of support interactions (known internally as contacts) being raised by Uber customers every week, our goal is to resolve these contacts within a predefined service level agreement (SLA). Contacts created by customers are resolved either via automation or with help from a customer support agent.

📖┆The 14 pains of building your own billing system

✍ Arnon Shimoni

I’ve seen them likened to an octopus, and I fully agree. They touch finance, product, experience, customer support, customers, legal, compliance, sales, and sometimes more.

✏ Data

The one thing that this job has taught me is that truth is stranger than fiction. — Predestination (2014)

📖┆"Good Enough" Data Models

✍ Joe Reis

Data modeling along the spectrum of perfect vs "good enough"

📖┆The Root Cause of All Problems in Data - Revisited

✍ Ergest Xheblati

What about a people and process problem? What does that mean? Does it have to do with people’s unwillingness to adopt data driven decisions or do we lack a good methodology for being data driven?

📖┆Data Quality within Lakehouses

✍ Piethein Strengholt

A deep dive into data quality using bronze, silver, and gold layered architectures.

📖┆How to measure a data platform?

✍ Timo Dechau

Product analytics for data products

📖┆dbt’s Model Groups & Access for Dummies

✍ Faith Lierheimer

Welcome to another edition of “Faith takes you on the ride while she learns how to do her job.” Today, we learn about a handful of dbt’s model governance features.

📖┆Making friends with the truth

✍ Jason Ganz

So how can we become data driven?

🤖 AI┆ML┆Data Science

You know, Burke, I don’t know which species is worse. — Ripley, Aliens (1986)

📖┆Supporting Diverse ML Systems at Netflix

✍ Netflix Technology Blog

In this article, we cover a few key integrations that we provide for various layers of the Metaflow stack at Netflix, as illustrated above. We will also showcase real-life ML projects that rely on them, to give an idea of the breadth of projects we support. Note that all projects leverage multiple integrations, but we highlight them in the context of the integration that they use most prominently. Importantly, all the use cases were engineered by practitioners themselves.

📖┆The Inevitability of AI in Every Facet of Your Business: 11 Deep Thoughts With Lasting Impact

✍ Sven Balnojan PhD

I want to share a collection of thoughts, quote from leaders in data & AI, and a few thoughts I have on them, to get you thinking. I have no clear answer to any of the questions they raise, and I keep coming back to them, revising my opinion over time.

📖┆Risk-Aware Product Decisions in A/B Tests with Multiple Metrics

✍ Mårten Schultzberg + Sebastian Ankargren + Mattias Frånberg

We summarize the findings in our recent paper, Schultzberg, Ankargren, and Frånberg (2024), where we explain how Spotify’s decision-making engine works and how the results of multiple metrics in an A/B test are combined into a single product decision.

📖┆User Action Sequence Modeling for Pinterest Ads Engagement Modeling

✍ Pinterest Engineering

In this blog post, we will mainly discuss how we adopt the user sequence features and the followup optimization:-Designed the sequence features-Leveraged Transformer for sequence modeling- Improved the serving efficiency by half precision inference. We will also share how to improve the model stability by Resilient Batch Norm.

📖┆Think Vs Compute (Part 2)

✍ Koo Ping Shung

So in this issue, this is a refinement further on what is thinking vs compute over here, or at least the differences between humans and machines. Hopefully by understanding the current differences I can work on how to incorporate them together, moving towards the vision of machine+human.

📖┆Evolving from Rule-based Classifier: Machine Learning Powered Auto Remediation in Netflix Data Platform

✍ Netflix Technology Blog

This is the first of the series of our work at Netflix on leveraging data insights and Machine Learning (ML) to improve the operational automation around the performance and cost efficiency of big data jobs.

📖┆What Is Trustworthy AI?

✍ Nikki Pope

Trustworthy AI is an approach to AI development that prioritizes safety and transparency for the people who interact with it.

📖┆Data governance in the age of generative AI

✍ Krishna Rupanagunta + Raghvender Arni + Imtiaz Sayed

In this post, we discuss the data governance needs of generative AI application data pipelines, a critical building block to govern data used by LLMs to improve the accuracy and relevance of their responses to user prompts in a safe, secure, and transparent manner. Enterprises are doing this by using proprietary data with approaches like Retrieval Augmented Generation (RAG), fine-tuning, and continued pre-training with foundation models.

🔥 Catch up

…Next Saturday night, we're sending you back to the future! — Dr. Emmett Brown, Back to the Future (1985)

📖┆BigQuery┆The INFORMATION_SCHEMA.WRITE_API_TIMELINE* views, containing per minute aggregated BigQuery Storage Write API ingestion statistics, are GA.

📖┆BigQueryDuet AI in BigQuery can now assist with Python code generation and code completion.

💠 Previously on Dimension

Dimension is my sub-newsletter where I note down things I learn from people smarter than me in the data engineering field.

Here are the 3 latest articles:

Published on 2024, February 24:

Published on 2024, March 2:

Published on 2024, March 9:

Not subscribed yet? Here you go:

Subscribe now

Get more from Vu Trinh in the Substack app

Available for iOS and Android

Get the app

👋 Hi, my name is Vu Trinh, a data engineer.

I enjoy reading good stuff (related to data and engineering), and this newsletter is my effort on the journey to seek the "good stuff" across the entire Internet.

Hope this issue find you well.

Referral Program

I’m launching a referral program to grow the community by giving you guys valuable gifts whenever you reach a referral milestone. The condition is simple: you refer friends to subscribe to my newsletter, and you will receive a gift based on the number of friends you refer. Here are the reward milestones:

Now, let’s refer friends and claim exciting rewards ;)

Refer a friend

📈 Career

Don't let comfort hold you back.

📖┆Career Pathways of Data Engineers

✍ Ravi Ganta

The career progression for Data Engineers is not a linear journey, unlike what might be expected. A straightforward trajectory, similar to that of Software Engineers ascending from entry-level individual contributors to executive leadership roles, does not necessarily apply to Data Engineers. This discourse delves into the intricate pathways that Data Engineers can undertake as they advance in their careers.

📖┆How to go from senior to staff data engineer in big tech

✍ Zach Wilson

In big tech, less than 3% of engineers make it to the staff level! If you want to beat the odds and actually become an L6+ engineer, this newsletter is for you. In my time in big tech, I went from junior to staff data engineer in four years. I’ll unveil the learnings I have from that journey.

🚀 Engineering

I have to believe in a world outside my own mind. — Memento (2000)

📖┆From Samza to Flink: A Decade of Stream Processing

✍ Chris Riccomini

I started Apache Samza twelve years ago during my tenure at LinkedIn. Samza was a stream processing framework built for Apache Kafka. The team grew to include all-stars like Martin Kleppmann, Chinmay Soman, Jakob Homan, Yi Pan, and many other talented engineers. Together, we added support for stateful processing, batch processing, SQL, YARN, standalone deployment, and many other features you see in modern stream processing systems. I learned a lot building Samza. In this post, I want to review Samza’s history, look at lessons learned, and talk about how these lessons affect my thinking on Apache Flink.

😉 Two-parts article:

Part 1: 📖┆Performance Improvements for Stateful Pipelines in Apache Spark Structured Streaming

✍ Mojgan Mazouchi + Mrityunjay Kumar + Anish Shrigondekar + Karthikeyan Ramasamy

Apache Spark™ Structured Streaming is a popular open-source stream processing platform that provides scalability and fault tolerance, built on top of the Spark SQL engine. Most incremental and streaming workloads on the Databricks Lakehouse Platform are powered by Structured Streaming, including Delta Live Tables and Auto Loader.

Part 2: 📖┆A Deep Dive into the Latest Performance Improvements of Stateful Pipelines in Apache Spark Structured Streaming

✍ Mojgan Mazouchi + Mrityunjay Kumar + Anish Shrigondekar + Karthikeyan Ramasamy

In this section, we will dig deeper into the various issues we observed while analyzing performance and outline specific enhancements we have implemented to address those issues.

📖┆Everything Ops: Beyond the Hype

✍ Benoit Pimpaud

Integrating “Ops” into our job titles and conceptual terms may come across as trendy, and indeed, it is.

📖┆lakeFS: Where’s my data?

✍ Ariel Shaqed (Scolnicov)

For a data management platform, sometimes it may seem as though lakeFS takes pains to hide your data. Indeed, one very common question on our Slack #help channel is a polite variation on “where’s my data?”. lakeFS does indeed keep your data and it does so inside your namespace. This core functionality works and is well-tested. But lakeFS does such a great job of abstracting away all of these details that it’s hard to find the data!

📖┆Introducing DoorDash’s In-House Search Engine

✍ Konstantin Shulgin + Satish Subhashrao Saley + Anish Walawalkar

We decided the best way to address these challenges was to move away from Elasticsearch to a homegrown search engine. We chose Apache Lucene as the core of the new search engine. The Search Engine uses a segment-replication model and separates indexing and searching traffic. We designed the index to store multiple types of documents with relations between them. Following the migration to DoorDash’s Search Engine, we saw a 50% p99.9 latency reduction and a 75% hardware cost decrease.

📖┆Query Builder Package: From Individual Chaos to Collective Consistency

✍ Sinem Elif Haseki

In the dynamic world of software development, aligning multiple products with consistent and efficient service request protocols is a critical challenge. This article delves into the role of a new query builder package in revolutionizing this process, emphasizing its necessity, benefits, and drawbacks.

📖┆How DotSlash makes executable deployment simpler

✍ Pascal Hartig

Andres Suarez and Michael Bolin, two software engineers at Meta, join Pascal Hartig (@passy) on the Meta Tech Podcast to discuss the ins and outs of DotSlash, a new open source tool from Meta.

📖┆We Need a Data Engineering-Specific Language

✍ Julien Hurault

In this article, I explore the present state of standardization within the industry.

✏ Data

The one thing that this job has taught me is that truth is stranger than fiction. — Predestination (2014)

📖┆Designing Metrics Trees

✍ Ergest Xheblati

A metric tree is a logical representation of a business’ growth model in a graph form. It’s a simplified representation of how inputs flow into outputs.

📖┆How the New York Times Games Data Team Revamped Its Reporting

✍ CJ Robinson

Ultimately, our goal is to make the data generation, ingestion, processing, and reporting pipeline as effortless and evergreen as possible.

📖┆10 Common Data Visualization Mistakes and How to Avoid Them

✍ Agoda Engineering

When creating data visualizations, it can be easy to make mistakes that lead to wrong interpretations. This article will look at bad data visualization and how to avoid it.

📖┆When a Data Mesh Doesn’t Make Sense for Your Organization

✍ Barr Moses

In this article, we’ll revisit data mesh to discuss what it is, why it makes sense for some teams, and when it doesn’t make sense for yours!

📖┆dbt’s Semantic Layer for Dummies

✍ Faith Lierheimer

What’s dbt’s Semantic Layer (powered by MetricFlow baby) supposed to do anyways?

🤖 AI┆ML┆Data Science

You know, Burke, I don’t know which species is worse. — Ripley, Aliens (1986)

📖┆2023 in Review: Recapping the Post-ChatGPT Era and What to Expect for 2024

✍ Leonie Monigatti

How the LLMOps landscape has evolved and why we haven’t seen many Generative AI applications in the wild yet — but maybe in 2024.

📖┆Happy New Year: GPT in 500 lines of SQL

✍ Quassnoi

It just proves that if you want something done right, you have to do it yourself. Encouraged by this optimistic forecast, today we will implement a large language model in SQL.

📖┆Meta's new LLM-based test generator is a sneak peek to the future of development

✍ Engineer’s Codex

Meta's TestGen-LLM is a sneak peek to the future of developer productivity: specialized, orchestrated, and rigorously filtered.

🔥 Catch up

…Next Saturday night, we're sending you back to the future! — Dr. Emmett Brown, Back to the Future (1985)

📖┆Introducing Apache Kafka 3.7

📖┆Copilot in PowerBI

📖┆PyAirbyte: Airbyte’s New Python Library for Moving Data.

📖┆The following BigQuery cross-cloud features are now generally available (GA):

• You can take advantage of the benefits of materialized views over Amazon S3 metadata cache-enabled BigLake tables.

• You can create materialized view replicas of materialized views over Amazon S3 metadata cache-enabled Biglake tables. Materialized view replicas let you use the materialized view data in queries while avoiding data egress costs and improving query performance.

• You can get information about materialized view replicas by using SQL, the bq command-line tool, or the BigQuery API.

• You can use cross-cloud joins to run queries that span both Google Cloud and BigQuery Omni regions.

📖┆BigQuery Materialized views can now reference logical views. This feature is in preview.

📖┆BigQuery GROUP BY ALL clause, which groups rows by inferring grouping keys from the SELECT items, is now in preview

💠 Previously on Dimension

Dimension is my sub-newsletter where I note down things I learn from people smarter than me in the data engineering field.

Here are the 3 latest articles:

Published on 2024, February 17:

Published on 2024, February 24:

Published on 2024, March 2: