I spent 8 hours learning about the Spark Out-Of-Memory (OOM) errors
What actually causes them and how to fix them
With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a student with an education email, use this 50% ANNUAL DISCOUNT
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.
Intro
If you’ve ever run Spark in production, you might encounter the OOM error once.
You might simply increase the executor memory, and the problem will be fixed. However, naively allocating more resources to your Spark job won’t be sustainable in the long term.
Instead, understanding the nature of the OOM is the better approach.
In this article, I deliver my understanding of the OOM errors so you can operate Spark more robustly in production.
Note 1: This article assumes you have a basic understanding of Spark. I highly recommend you read this article for that purpose.
Note 2: This article only discusses OOM on Spark executors.
How Spark works?
In brief
If you want to run Spark, you must have a cluster of machines that provide the resources for the Spark cluster.
A Spark cluster is a set of JVM processes, including a Driver and Executors. Those processes run on the cluster of machines (with communication with the Cluster Manager).
Every Spark cluster is associated with a Spark application.
Below the application is the Spark job. A job represents a series of transformations applied to data: the entire workflow from start to finish. The series of transformations (e.g, filter, map…) can be triggered only by an action (e.g., show, count,…). We can say that a job is associated with an action. An application can have multiple jobs.
A job is split into different stages when a transformation requires shuffling data across partitions. (e.g., groupBy, join). A stage is a job segment executed without data shuffling.
A stage has a set of tasks. A task is the smallest unit of execution within Spark. Each stage is divided into multiple tasks, each handling a partition, a portion of data from an external source or from the upstream stage.
At a given stage, tasks can run in parallel; the parallelism depends on the executor's CPUs. You can understand that tasks are handled in parallel in an executor using the multithreading paradigm. By default, a task is handled by an executor core (controlled by the “spark.task.cpus” setting); if the executor has 4 cores, 4 tasks can run in parallel within the executor.
Why do OOM errors happen?
With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a student with an education email, use this 50% ANNUAL DISCOUNT
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.
