[FREE] Solve higher-impact problems
The mindset to grow and get promoted.
Reminder: I’m offering a limited-time 50% discount on the annual plan:
Once you claim it, the discount will be applied forever.
Now, with only $5/month, you will have access to:
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues
Intro
I’ve had the chance to work with distinguished senior colleagues. They’re distinguished not because they know a lot of technical stuff (although technical knowledge is a factor).
They try to solve higher-impact problems.
You won’t find any illustrations (except for the landing image) like in my previous articles; that is intentional. I want your 100% focus on the text because things I will share with you are the one I wished someone tell me sooner.
You might also notice that I've repeated ideas I've discussed in my previous articles. I believe important things should be repeated over and over again.
Value
Companies hire us (the DEs) to help build a robust data foundation so others can leverage that to extract business insights.
Although the details of job descriptions differ, the ultimate goal will always be the same: store, process, manage, and serve data efficiently — the data foundation.
Any company hires an employee, assuming he is good at “X,” so he can help the company with that. For data engineers, we are hired because we’re good at “data engineering,” and the company believes we can use that to build their data foundation.
You’re not hired solely for your ability to write Spark jobs, debug Airflow errors, or model a star schema. You’re hired because the company believes you can leverage all those things to help them store, process, manage, and serve data.
Thus, every single task you do and every decision you make must deliver some business value (directly or indirectly).
Business value must be the number one priority.
To answer what value you could help create, you need to be aware of the true responsibility of a data engineer. Joe Reis and Matt Housley define this very well in the Fundamentals of Data Engineering: Plan and Build Robust Data Systems:
Data engineering is the development, implementation, and maintenance of systems and processes that take in raw data and produce high-quality, consistent information that supports downstream use cases, such as analysis and machine learning. Data engineering is the intersection of security, data management, DataOps, data architecture, orchestration, and software engineering.
A data engineer manages the data engineering lifecycle, beginning with getting data from source systems and ending with serving data for use cases, such as analysis or machine learning
— Joe Reis, Matt Housley.
However, simply putting all that information into your mind is not enough. Observe what you’re doing and what others do in your company; try to figure out your responsibility yourself, or consult the manager or senior colleagues.
At the end of the day, your responsibilities come down to what Joe Reis and Matt Housley defined. However, figuring them out on your own makes you much more aware of them.
The more business-value-oriented you are, the more chances you have to grow and contribute. A signal that tells you you’re on the right path: you focus more on the “boring“ things: data modeling, security, or governance.
Higher-impact problems
Reminder: I’m offering a limited-time 50% discount on the annual plan:
Once you claim it, the discount will be applied forever.
Now, with only $5/month, you will have access to:
200+ deep-dive data engineering articles
practice-spark: 65 LeetCode-style problems to practice Spark SQL/DataFrame
learn-spark/dbt/airflow: CLI tools to master Spark/dbt/Airflow
If you’re a Vietnamese user, please DM me for an upgrade due to payment issues
When you understand that your responsibility and business value are the 1st priority, you will start reflecting on what you’re doing.
—
I used to measure my success by the number of tasks I finished. The more tasks I do in a day, the better I feel.
Finishing tasks matters.
It isn’t wrong; you’re even praised and paid for it.
But you have to slow down, observe, and start looking for work that matters… more.
—
When you start your career, the world is small.
An Airflow task error, a Python script that’s not working, or a slow Spark job.
You are standing on the ground, ensuring every single detail works as expected.
However, to grow and promote, you need to go “higher” to understand:
Why you’re doing this: Some tasks add value; others don’t.
The bigger picture: you’re not just coding a pipeline that someone else designed forever. You have to step up to design, plan, and be responsible for work that matters more.
When looking at the higher place, the problem isn’t always as straightforward as it appears in your Jira ticket. It will be more vague:
Why don’t users trust our data?
Finance says revenue is $1M, while marketing says it’s $1.2M — both pulled from the data warehouse. Why is there a discrepancy?
Is there a way to improve the productivity of Spark and Airflow development on our team?
It’s vaguer than ‘how to run your Spark job’ because the problems occur at a larger scale. They no longer live only in your IDE; they involve your whole team, other teams, or even the entire company.
This explains why solving these problems creates more impact. And creating impact is good for your growth (and reputation)
When it involves more people, you have to communicate, and human language is not binary like your code; it’s fuzzy. That makes understanding and solving the problem even harder.
—
What I'm trying to say is that every task and every problem matters. Solving a Jira ticket is valuable, but some work matters more—the kind of work that has a much larger impact.
Outro
Here is the summary:
Having a mindset that prioritizes business value will motivate you to seek higher-impact problems.