To celebrate Lunar New Year (the true New Year holiday in Vietnam), I’m offering 50% off the annual subscription. The offer ends soon; grab it now to get full access to nearly 200 high-quality data engineering articles.
Intro
Parquet has become the standard file format in modern data analytics, thanks to its efficiency in both storage and query performance.
Everybody knows about Parquet's columnar layout, but few know how data is physically stored, especially how it is encoded and compressed. In this article, I will provide a deep dive into Parquet’s encoding capabilities, from its type system to how data is represented based on each type.
The row groups and column chunks
The Parquet format organizes data using the Partition Attributes Across (PAX) layout, commonly referred to as the hybrid format. It first groups data into “row groups,” each containing a subset of rows. (horizontal partition.)
Within each row group, data is stored column by column; values from a column are stored together. Each row group’s column is called the column chunk. Each chunk is composed of pages, which are the unit for encoding and compression.
Because data from the same column are stored back-to-back within a chunk, values of the same type remain close together. In addition, data in the same column tend to be more homogeneous and repetitive, which significantly benefits the encoding and compression processes, as these rely on data patterns.
Before diving into these processes, we will first explore Parquet’s type system, as the input column’s type largely impacts the encoding and compression schemes.
Type systems
Parquet distinguishes between logical and physical types:
Logical Type: What the data semantically means.
Physical Type: How the data is physically stored on disk.
Physical
These are the only few types Parquet actually knows how to write to a file. The creator keeps the options small to make the implementation of the Parquet reader and writer simpler:
BOOLEAN: A single-bit true/false value.INT32: A 32-bit signed integer.INT64: A 64-bit signed integer.FLOAT: A 32-bit floating-point number.DOUBLE: A 64-bit floating-point number.BYTE_ARRAY: A variable-length array of raw bytes.FIXED_LEN_BYTE_ARRAY: A fixed-length array of raw bytes.INT96: (Deprecated) A 96-bit integer, primarily used for legacy timestamp formats.
The encoding and compression only work with physical types.
Logical
However, your application might need to be represented with a richer type system. Parquet supports logical types, which are wrappers around physical types with additional metadata to help the engine interpret the bytes correctly.
For example, a STRING is stored as a BYTE_ARRAY; the raw bytes are interpreted as a UTF-8-encoded string. A DATE logical type annotates an INT32 that stores the number of days from the Unix epoch.
Encoding scheme
PLAIN
This scheme is the most straightforward. It serializes values back-to-back in a standardized, little-endian binary format with no complex transformations. PLAIN must be supported for all physical types defined in the Parquet specification.
BOOLEAN: 1 bit per value, zero is false; one is true.
INT32: 4 bytes each value.
INT64: 8 bytes each value.
FLOAT: 4 bytes each value.
DOUBLE: 8 bytes each value.
BYTE_ARRAY: The length is stored in 4 bytes, followed by bytes.
FIXED_LEN_BYTE_ARRAY: it simply stored as bytes
This encoding is the default choice when no other scheme offers a clear advantage. It is best suited for data that lacks patterns, such as columns with high cardinality (many unique values), randomness, or unpredictability.
RLE_DICTIONARY
This one is one of Parquet’s most commonly used encodings. The writer first scans the data in a column chunk to build a “dictionary” of all unique values. This dictionary is stored once in a dedicated dictionary page. This page is encoded using the PLAIN scheme.
To celebrate Lunar New Year (the true New Year holiday in Vietnam), I’m offering 50% off the annual subscription. The offer ends soon; grab it now to get full access to nearly 200 high-quality data engineering articles.
