A 9-minute simple explanation of Spark Shuffle
What is it, when it happens, the details of the shuffle process behind the scenes and how can we optimize it
With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a student with an education email, use this 50% ANNUAL DISCOUNT
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.
Intro
Since its first release, Apache Spark has established itself as a leader in data processing.
It’s fast
It has a whole ecosystem (batch, stream, ML, …)
It has many configuration knobs you can tune
And it provides a wide range of interfaces: SQL, Python, Scala, Java, R…
Most people believe Spark efficiency comes from in-memory processing. That’s true, but not enough, as Spark's performance is influenced by many factors, including query planning, lazy and adaptive execution, memory management, cache management, and, especially, shuffle-based (MapReduce-style) processing.
In this week's article, I tried my best to dive into this aspect of Spark: what shuffle is, when it happens, the details of the shuffle process behind the scenes, and how we can optimize it.
Narrow vs Wide transformation
Spark introduced the RDD abstraction to manage data in memory. All other abstractions, such as DataFrame to Dataset, are compiled into RDDs behind the scenes. Each RDD has a list of partitions, each of which is a subset of your data.
When you define an RDD, its data is unavailable or is transformed immediately until an action triggers execution. This approach allows Spark to determine the most efficient way to execute the transformations:
Transformations, such as
maporfilter, define how the data should be transformed, but they don’t execute until an action forces the computation. Because RDDs are immutable, Spark creates a new RDD after a transformation.Actions are the commands that Spark runs to produce output or store data (e.g.,
collect), thereby driving the actual execution of the transformations.
A Spark job associated with actions is split into stages during planning. Each stage has multiple tasks, which process data partitions in parallel.
Speaking of stages, the stage boundary only appears when data shuffling appears (which is also today's main topic), and data is shuffled when there are wide-dependency transformations:
Transformations with narrow dependencies are those where each partition in the downstream RDD has a limited number of dependencies on partitions in the parent RDD. These partitions may depend on a single parent (e.g., the map operator) or on a specific subset of parent partitions known beforehand (e.g., with coalesce).
Transformations with wide dependencies require data to be partitioned in a specific way, where a single partition of an upstream RDD contributes to multiple partitions of the child RDD. This typically occurs with operations such as groupByKey or join, which involve data shuffling.
These operators require data with similar attributes (e.g., join keys or aggregation columns) to be processed on the same worker. This process is called data shuffling.
Consequently, wide dependencies result in stage boundaries; tasks in the child stage create new data partitions with data with similar attributes.
We can understand that narrow dependency transformations can be “pipelined” and handled in a stage, reducing network communication overhead; on the other hand, wide dependency transformations are more expensive.
Shuffle behind the scenes.
But why are wide dependency transformations, especially the shuffling process, expensive?
Naming Convention
For the naming convention, I use “mappers“ to refer to the tasks that output the data in the shuffling process, and “reducers” to refer to tasks that read data from the mappers.
This convention is widely used in MapReduce; the paradigm has brought the data shuffling concept closer to the community. However, as I understand it, MapReduce doesn’t invent data shuffling, since this process has long existed in DBMSs.
Setup
Imagine we have a Spark job with two transformations: first, the data is filtered, and then it is grouped by column X to calculate the sum.
Mappers
After planning and scheduling, we will have two Spark stages. The first filters the data, and the second groups the data by the value of column X.
In the first stage, a set of tasks runs in parallel, reading data from the source and filtering the data.
The data is first read into memory as RDDs with a given number of partitions.
The number of partitions may be determined by your source data (e.g., a Parquet file), or you can determine it yourself. This number is important because it defines resource utilization and the level of parallelism. Spark suggests 2-3 tasks per CPU core in the Spark cluster.
For example, if you have 4 executors with 4 CPU cores each, the total cores will be 16, and the number of partitions (as one task is in charge of one partition) should be 32-48 (given the “spark.task.cpus” is 1, which means 1 task has 1 CPU core).
After the data is loaded into memory, the input filter logic is applied to it.
To prepare for the next stage, the shuffle process must happen to bring records with the same column x value to the same partition. Mappers (the filter tasks) will write filtered records to shuffled partitions on disks.
Yeah, you heard it right: to disk, not memory, as people often misunderstand because Spark is famous for in-memory processing.
With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a student with an education email, use this 50% ANNUAL DISCOUNT
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.
