Docker for data engineer
Containerization → Reproducibility + Isolation
I'm offering a limited-time 50% discount on the annual plan:
Once you claim it, the discount will be applied forever.
Now, with only $5/month, you will have access to:
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues
Intro
I’ve had the chance to learn so many things through my career. From data engineering stuff like BigQuery, Spark, Airflow, and Dagster, to actually developing a backend application or a CI/CD pipeline.
I feel lucky.
However, one piece of knowledge I truly appreciate having gained early.
It’s Docker—or, more generally, the ability to containerize any application.
To continue my process of unlearning (first with Parquet here), I relearned Docker and am sharing what I discovered in this article.
The goal is not to cover every corner of Docker; instead, I aim to provide enough material to help you understand and apply what you learn here to real-world workloads.
Note: What I want to cover here is primarily containerization. However, there is currently no containerization technology more dominant than Docker, so most of this article still focuses on Docker. Whenever the opportunity arises, I'll zoom out and discuss containerization concepts more broadly.
What is it?
In the past, if a company wanted to run a new application, the IT department needed to buy a new server. Then, the developer would deploy the application on it.
However, the downside is that a server back then could only handle a single application; if another app required an isolated runtime, it must be run on a different server.
Predicting the application’s utilization is indeed a challenge: if the server doesn’t give enough resources, you must buy more hardware; if the application doesn’t use all the resources, you accept the waste, as a server can only handle one isolated application at a time.
Virtual machines fixed this.
With a hypervisor, one physical server can host multiple applications, each in its own VM, securely and in isolation.
It was a big deal. (You could imagine)
However, VMs have overhead.
Every VM needs its own full operating system, and that OS consumes CPU and RAM that could have been used for your application. The OS needs patching, monitoring, and sometimes a license fee.
Then, a container is introduced to solve the problem.
From the outside, a container looks like a VM. Your application still runs in isolation from everything else.
The difference is that it doesn’t bring its own OS; all containers on a machine share the host’s kernel. No separate OS means no OS overhead: the application can use more RAM and CPU now, and the container's startup time will be faster than a VM's.
We can’t talk about containers without mentioning Docker.
Container technology existed long before Docker. However, it was complex and not so user-friendly at first.
It wasn’t until Docker came into town. When you hear people mention “Docker”, there is a high chance they are referring to the Docker technology, not the company Docker, Inc.
Docker is the software that creates, manages, and orchestrates containers. It’s built from various tools from the Moby open-source project. Docker allows users to focus on applications without worrying much about the underlying infrastructure.
For the rest of the article, I will use “Docker“ to refer to the Docker software, not the Docker, Inc company.
Before diving into it, let’s clarify one thing: Why should data engineers learn Docker, or more generally, containerization?
Why learn Docker?
September 23rd, 2019. My first day at my first (data) job.
I had just graduated, and I walked in expecting to touch the cool stuff: Spark, Hadoop, Kafka, etc.
But life is life; nothing as expected.
My first task was to do something I had never heard of: Docker. I struggled for two weeks. Now, looking back, I realize the main reason for the struggle was skipping the “why” and jumping straight into remembering the commands and getting things running.
Almost seven years later, I can say that Docker has quietly become one of the most important skills in my career. It does nothing to make my applications, queries, or data transformations faster, but it provides software characteristics that are just as important as performance: reproducibility and isolation.
Having these two means you can do a lot of things in data/software engineering:
I’m offering a limited-time 50% discount on the annual plan:
Once you claim it, the discount will be applied forever.
Grab it now and become a paid member to get access to:
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues