10 Minutes to Learn Apache Spark JOINs with a Hands-On Project
How do JOIN operations work in Spark?
With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a student with an education email, use this 50% ANNUAL DISCOUNT
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.
Intro
Apache Spark was designed to solve the problem of processing large volumes of data during data movement from A to B.
It has evolved.
In 2015, the creator introduced the new Spark SQL model, which let users leverage the benefits of relational processing. That means the user can express the processing logic using “SELECT“, “GROUP BY“, “SUM“, “AVG“, and especially “JOIN”.
The ability to work with multiple datasets is the backbone of many business transformations.
JOINs are among the most commonly used operations in Spark. Although I shared how Spark JOINs work here, it is still theories.
We need to validate them, as I did with Spark’s fundamentals here.
In this article, we will walk through a simple project with (a quite dummy) Spark JOIN logic to understand what actually happens behind the scenes.
Prerequisite: I expect you to have some Spark fundamentals, such as Spark clusters, Spark jobs/stages/tasks, or data shuffling. You can read my Spark series here.
Prepare the data
First, clone this repo and enter that repo for the rest of the project.
Then, install the Python packages:
pip install -r requirements.txtFor this project, we will join two datasets from the TPC-H: the “lineitem” (~2.6GB) and the “order” (~600MB) using the “o_orderkey” column. To generate these two tables:
cd data && bench tpch gen -s 10The logic
The processing logic is in the .data/join.py file. It simply reads these two datasets and joins them together.
For the Spark cluster resource:
There will be 2 executor instances. Each will have 2 cores and 4GB RAM. This means the application’s parallelism is 4, where a task could be handled by a CPU core (controlled by spark.task.cpus, which is 1 by default)
We will let the driver’s resource be the default.
All the configurations related to the “eventLog” are simply used for logging.
We ignore dynamic resource allocation here to monitor the application’s resource easier.
The “spark.sql.files.maxPartitionBytes” is 256 MB, which means Spark will try to pack no more than 256MB of Parquet data into a partition when reading from files. The higher the “spark.sql.files.maxPartitionBytes”, the larger the partition.
The Spark Standalone Cluster
We need a cluster of machines to launch our Spark cluster. In this project, we start a Spark Standalone cluster in Docker:
docker compose up -d
Docker will pull the official Spark 4.0.0 image and run scripts to start a master container (exposed on port 8080), a history server (for debugging, exposed on port 18080), and two workers (each with 4 CPU cores and 4 GB of RAM).
Sort Merge Join (SMJ)
After the done with the preparation, let’s submit our app:
docker exec -it spark-master /opt/spark/bin/spark-submit --master spark://spark-master:7077 /opt/spark/work-dir/spark-warehouse/data/join.pyAfter the application completes, we can visit the Spark History server (localhost:18080) to see what actually happened. In this article, we focus only on the JOIN operation. For other details, such as the input stage, partition size, and parallelism, please see my previous article.
—
For the physical execution plan, we can choose the just finished application, switch to the “SQL/Dataframe“ tab, choose the completed query, and click “Details “
For our inner join operation, we see that Spark applied the SMJ for us.
Note: SortMergeJoin is not an exclusive technique in Spark; it has long existed in the database field.
With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a student with an education email, use this 50% ANNUAL DISCOUNT
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.
