Polars at a high level
An OLAP query engine with DataFrame API
With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a student with an education email, use this 50% ANNUAL DISCOUNT
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.
If you'd like more articles on Polars, please give a like or comment to let me know.
Intro
When it comes to the single-node processing engine, there are two strong candidates here: DuckDB and Polars.
Last time, we learned about DuckDB, and I think we should also bring Polars into the discussion to gain a more comprehensive understanding of these two popular single-node data processing systems.
In this article, we examine what Polars is, its motivation, design decisions, and use cases.
Note: Although Polars support distributed processing, this article only discusses on its original form: a single-node data processing engine.
Motivation
In 2020, Ritchie Vink was stuck at home because of the COVID pandemic. He spent time on some Rust pet projects. A few were done. But he had an itch. He observed that Rust still lacked a proper DataFrame library (as in Python with Pandas). So he tried something small: a simple Rust crate that does that. (In Rust, a crate is a unit of compilation)
From there, things got out of hand (in a positive way).
Less than a year later, Polars 0.6.5 was already beating Spark and pandas by 4 to 6x on the H2O.ai groupby and join benchmarks.
The motivation for Polars also stemmed from the hardware: Vendors kept adding cores, but tools like NumPy and Pandas still had to be single-threaded (because of Python’s design). Ritchie observed that his own laptop had 12 cores and that Pandas was using less than 10% of the available compute.
Then Polars stopped being a side project.
In 2023, Vink and Chiel Peters, colleagues for five years by then, incorporated Polars and raised $4M. At the time, Polars had around 6 million total downloads and 20K GitHub stars. And when I checked the download statistics for the last 30 days, it's nearly 60 million.
That’s how Polars was created: an initial pet project to combine the famous data manipulation interface (DataFrame), the high-performance language (Rust), and the implementation of decades of database research: logical plans, cost-based rewriting, predicate pushdown, plus all other things.
That is indeed the product-market fit in which data professionals rely heavily on the DataFrame API, but the performance of Pandas and NumPy remains a serious bottleneck.
How It Actually Works
With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a student with an education email, use this 50% ANNUAL DISCOUNT
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.