I've been blogging for years at articles.Analytics.Today, and it's terrific to find a fellow tech enthusiast with a real passion and skill for writing. Superb post. I'll include a link to this in my next article - on how Snowflake works with Apache Iceberg over Parquet. Well done!
One question: in the file metadata, we have the row group metadata and the root group metadata. We have the data page offset or the dictionary page offset, and we also store the min and max values of the columns. What does the column metadata hold, and where does the column metadata live?
I've been blogging for years at articles.Analytics.Today, and it's terrific to find a fellow tech enthusiast with a real passion and skill for writing. Superb post. I'll include a link to this in my next article - on how Snowflake works with Apache Iceberg over Parquet. Well done!
Great sharing!!
Truly awesome post, explaining in detail about parquet. Thanks a million for sharing.
Good Parquet overview!
Very goos article
Great article (as always)!
Thank you for this great article !
Very detailed.
Very detailed overview for someone who works with parquet and spark sql at work.
Bookmarking this
Great article!
I think there is a tiny mistake in the last picture:
For row group 2 the system must read the yellow column chunk C, otherwise it wouldn't know if the condition C<10 is met.
Cheers Chris
hey Vu, great illustrations!
i do wanna add to the `Encoding` part a bit
after RLE, parquet also applies bit-packing to save spaces further, see https://parquet.apache.org/docs/file-format/data-pages/encodings/#run-length-encoding--bit-packing-hybrid-rle--3
it's not just solely RLE, but a combination of both encoding techniques
Great 👍
One question: in the file metadata, we have the row group metadata and the root group metadata. We have the data page offset or the dictionary page offset, and we also store the min and max values of the columns. What does the column metadata hold, and where does the column metadata live?
🙌🙌🙌
Nice work
well written article!