Here's everything you need to know about Parquet in 2026.
I spent 10 hours re-learning Parquet.
Reminder: I’m offering a limited-time 50% discount on the annual plan:
Once you claim it, the discount will be applied forever.
Now, with only $5/month, you will have access to:
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues
Intro
I caught this quote not long ago:
It is impossible for a man to learn what he thinks he already knows. — Epictetus
That quote has been living rent-free in my mind for days.
It was totally wrong to me at first.
On a Saturday morning, I decided to reflect on my journey, from having been a data engineer for 6 years to being the author of this newsletter for 3 years.
I’ve gradually come to understand the meaning behind that quote.
I’ve spent a lot of time researching and learning about data engineering. Over time, I started taking my knowledge for granted. I developed the illusion that I knew everything, making me arrogant and less receptive to new ideas (“I already know that”/“It’s just a new way to talk about an old idea“)
“Sh*t, that’s bad; that will make me stop learning someday“
—
Lately, I’ve been practicing unlearning something I know (or thought I knew). That process is harder than it sounds: forget everything you know about something and take the most naive look at it, again.
—
One of the first things I pick up is the one I've talked about endlessly in this newsletter: the Parquet file format.
That’s why I’m writing this article.
Columnar storage
In 2026, the understanding of “columnar storage” is not the same as it was in 2019.
Back then, when the term was mentioned, many people thought that each column’s data was stored separately (including me).
That’s true, but only for some cases.
Recently, thanks to many high-quality blog posts, video sharing (and AI), columnar storage has been increasingly well understood.
When a file format says it stores data in columnar fashion, you must dive deeper to see how the “columnar“ is actually implemented.
—
We know that, in most cases, row-wise format is good for OLTP and bad for OLAP.
And, column-wise is good for OLAP and bad for OLTP.
This is because the two serve different kinds of workload:
Point query vs. history scan: A backend database (e.g., PostgreSQL) usually serves queries that require only a few records at a time. A data warehouse service (e.g., BigQuery) usually serves queries that require scanning historical data with billions of rows.
Whole row vs. some columns: In an OLTP system, a query usually needs to retrieve the whole row to get all the information. In an OLAP system, when GROUP BY is used on some columns, bringing all columns into memory is unnecessary.
Fast write vs batch load: OLTP requires fast data writes to reflect changes in the new world as quickly as possible, since these OLTP databases are the state for many services that contribute to business flow.
For example, when a user subscribes to Netflix, the backend must persist that record so that when the user refreshes the UI, they immediately see they can access all the movies.
On the other hand, the dominant use case in data warehouses is batch loading; the cadence can be hourly, daily, or weekly.
Latest state vs. history data: People usually care about the latest state of the record in the OLTP world: latest balance, latest order, latest address. In OLAP, to gain insight, the latest state alone is not enough; historical data must be retained for days, months, or years; storage space will be much larger; data must be encoded/compressed to prevent your storage bill from exploding.
—
OLTP databases with row-wise formats store data from the same row close together. This means that if we have 5 records, record A is stored first, then record B, then record C, and so on.
This works well when the system accesses the entire record most of the time. With the help of indexing (e.g., BTree), the system can quickly locate a record; then, it simply loads the entire record into memory for consumption.
Organizing records right after one also makes writing fast. The system first locates an available slot for new data, then writes the entire record continuously without jumping around.
Thus, you know why the row-wise format fits OLTP workload.
—
However, the row-wise format doesn’t work well with OLAP.
The whole record and fast write benefits are not attractive here; when doing aggregation that needs 3 columns on billions of rows, with a row-wise format, the system must bring all column data into memory to extract 3 columns; with a table of 100 columns, you waste bandwidth loading 97 columns. When you multiply that by a billion rows, the waste and overhead are huge.
—
Then people started thinking about storing data in a columnar format: data from each column is stored continuously and separately. This reduces I/O costs, as you can load only 3-column data now, instead of the whole record as with row-wise loading. Also, data from the same column tends to be homogeneous and repetitive, which makes it more efficient to encode and compress.
—
However, storing data in that way has some downsides.
The most obvious is that the data writes require touching multiple column segments; writing 100 columns requires the system to jump around 100 places to write all the data for a single record.
In addition, when queries involve multiple columns, the system must reconstruct the records from separate segments. The cost of this reconstruction increases with the number of columns involved in the query.
—
The hybrid format steals the best from both worlds.
The format first splits the data horizontally into portions, each containing a subset of rows. In each portion, data for each column is stored next to each other.
This format still provides the benefit of the columnar format, while keeping data from the same row close together; it is, of course, not as close as in the row-wise format, but the row’s data is now kept in the same portion, reducing the cost of writing data and reconciling a record, as data from the same record is distributed only within the horizontal portion.
—
When a format claims to be columnar, it is, 80% of the time, a hybrid format. From the internal format of DuckDB, Snowflake, BigQuery, ORC, and the one we discuss in this article: Parquet.
A small number of formats, such as ClickHouse or Redshift's internal format, are purely columnar.
In the rest of this article, I will use the hybrid and columnar formats interchangeably.
—
Parquet first groups data into “row groups,” each containing a subset of rows.
Within each row group, values from a column are stored together. Each row group’s column is called the column chunk. A chunk has a set of pages.
A page is the smallest data unit in Parquet, used for encoding and compression. There are several types of pages, including data pages (which contain the actual data) or dictionary pages (which contain dictionary-encoded values).
Reminder: I’m offering a limited-time 50% discount on the annual plan:
Once you claim it, the discount will be applied forever.
Now, with only $5/month, you will have access to:
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues