A small hands-on project to 2× your Apache Spark learning process
Validate the theory yourself with the available code you can run along with
With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a student with an education email, use this 50% ANNUAL DISCOUNT
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.
Intro
I received some feedback that, although my Spark articles are very informative, they lack hands-on experience. Because of that, they usually forget what they read a few days later. I agree. The fastest way to learn something new is to learn enough theory, try it yourself, and validate your understanding.
I decided to do a project using Spark to process some data and share my notes in this article. We will process 20GB of data using Apache Spark.
I hear you. “Only 20GB? On Spark?“
My intention is much more than 1TB or even 10TB of data. Although we can now process “big data” with Spark, that volume of data barely fits on most laptops. Renting cloud virtual machines could be an option; however, with that volume of data, we need to request that the cloud vendor increase the VM’s CPU quota (free cloud trials don’t let you run a VM with many CPUs) and carefully manage costs.
So, 20GB of parquet data is more feasible.
The goal of this project is to have a closer look at how the data will be processed in a distributed manner in Spark. We also tune some configurations to adjust the parallelism and monitor what actually happens behind the scenes via the Spark UI.
Of course, the details delivered in this article won’t cover every single thing you need to tune your Spark production workload. However, I hope my work can clear the mist, boost your motivation (there is code so you can run along), and provide the fundamentals to handle your future Spark workload.
Disclaimer: The results I’m gonna show in this article come from the Spark application submissions on my laptop, and the results might be different when you run it, as the performance of the Spark application might be affected by the current status of the laptop, such as resource contention due to running other applications.
Based on my experience, if you want a more isolated setup, you could spin up a VM instance with 12 cores and 36 GB of RAM to experiment. That spec won’t cost you more than $1 per hour, and the cloud free credit trial could cover the VM cost.
tl;dr
You will be guided to prepare the 20GB of data, set up the Spark Standalone cluster via Docker, and submit the application.
A Spark job will be divided into multiple stages. A stage will have multiple tasks, which are the smallest unit of work in Spark. A task will handle a partition, a piece of data. Tasks could be run in parallel in an executor.
You can understand that tasks are handled in parallel in an executor using the multithreading paradigm.
Increasing the number of executor cores will increase the parallelism. As a task is executed on a defined number of cores (1 by default, controlled by the “spark.task.cpus” setting).
However, increasing executor cores but keeping the executor’s memory the same will shrink the memory portion of each task because more tasks now share the same memory pool.
Increasing the executor memory will give the task a larger memory portion, which helps reduce the chance of spilling data and improve the overall performance.
The memory portion used for the task’s processing and storage is not all the “spark.executor.memory”. It’s calculated by:
(spark.executor.memory - reserved memory (300MB in default) ) * spark.memory.fraction (0.6 default)
The size of the partition is important as it determines the workload for a single task and the total number of tasks:
The larger the partition, the smaller the number of partitions.
The time to handle a single task might be longer, and there is a higher chance that the task spills data to disk
Although a lower partition size might increase the number of partitions, it reduces the workload for a single task and lowers the chance of data spilling.
The data type affect the way Spark do aggregation. For example, if you aggregates String, Spark will choose to go with SortAggregate, which is as not as efficient as the HashAggregate.
Prepare data
For all the code and the logic, you can clone this repo and follow along.
Make sure you run the following commands inside the repo folder. First, install the Python packages:
pip install -r requirements.txtTo generate the data, we need the “initial“ data file so we can duplicate it until we get 20GB of data. This parquet data file (part of the tpc-ds dataset) will be generated via the ibis-bench library:
bench tpch gen -s 10Then we copy this file into the ./data folder. We will mount this folder into the Spark standalone cluster on Docker containers, more on this later.
cp ./tpch_data/parquet/sf=10/n=1/lineitem/0000.parquet ./data/0000.parquetThere is also the “duplicate.sh” script, which accepts the target data size as a parameter. The script will duplicate 0000.parquet to reach the target. To generate 20GB of parquet data, run this:
chmod +x duplicate.sh && ./duplicate.sh 20The processing logic is in the .data/main.py file.
It simply reads the folder of generated Parquet files and does simple aggregations. An important note: I tried to select and operate on all the fields in the Parquet files so we can actually process 20GB of data. Because Parquet is a columnar format (hybrid), selecting only a few columns will significantly reduce the data scan. However, we don’t want that in this project. We want a solid 20GB.
Start the cluster
Next, we will start the Standalone Spark Cluster via Docker.
This is not a Spark cluster.
This is a cluster of machines that provide resource for the Spark cluster, which is a set of JVM processes (driver and executors). The Spark cluster will ask the cluster manager resource from the cluster of machines. There are different managers, such as YARN, Kubernetes, or, in our case, the standalone one.
I prepared the “docker-compose.yaml“ file so you can simply run:
docker compose up -dFor the Spark master (the cluster manager), we use the official Spark 4.0.0 image, expose ports 8080 and 7077, set the necessary environment variables, and run the “start-master.sh” script to start the master process. I also mount the “data” and “spark-events” volumes into this Docker container. The first stores the PySpark file and the Parquet data; the latter stores monitoring information used by the Spark History server.
For the worker, we also use the same image, set the worker capacity in the environment variable, and run the script to start the worker process. We also mount the data and spark-events volumes for the worker.
The last component is the Spark History Server, where we can observe metrics from our Spark application. The service is exposed on port 18080.
Let’s run our application
Here is the PySpark logic. It simply initiates a Spark session, reads a folder of Parquet files, and performs a simple GroupBy that touches all the columns.
The resource request
Let’s walk through the SparkSession initialization code block:
With only $7/month (billed annually), you can access all the materials you need to grow from junior → senior DE.
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a student with an education email, use this 50% ANNUAL DISCOUNT
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues. As compensation for the inconvenience, you’ll get 50% OFF the annual plan.
