This is GroupBy, the place where I share with you guys the resources I learn from people smarter than me in data engineer field.
Not subscribed yet? Here you go:
👋 Hi, my name is Vu Trinh, a data engineer.
I enjoy reading good stuff (related to data and engineering), and this newsletter is my effort on the journey to seek the "good stuff" across the entire Internet.
Hope this issue find you well.
🎯 Side Project
40+ hours of debugging and you still want some more?
To get your hand dirty (more), this week I will bring you a project:
The goal of this project is to perform data analytics on Uber data using various tools and technologies, including GCP Storage, Python, Compute Instance, Mage Data Pipeline Tool, BigQuery, and Looker Studio.
dbt, a popular tool for abstraction transforming and modeling data.
Learning dbt is essential for streamlining data processes, ensuring data quality, and accelerating analytics development, making it a valuable skill for anyone involved in data analysis and management.
Learn with the analytics engineers of dbt Labs how to migrate legacy transformation code into modular dbt data models. Useful if you're porting stored procedures or SQL scripts into your dbt project. (approximately 3.5 hours)
Learn more about the theory of data testing and the practice of creating custom generic tests, leveraging tests in packages, and applying test configurations. (approximately 4 hours).
Approximately 16.5 hours for you to understand that “dbt is not just a SQL generator“
🚀 Engineering
Engineering is the practice of using natural science, mathematics, and the engineering design process to solve technical problems, increase efficiency and productivity, and improve systems. — wikipedia
In a distributed PubSub environment, complexities related to client-server communication can often be hard blockers for application developers, and solving them often require a joint investigation between the application and platform teams.
Without metadata, there are no discovery, collaboration, or quality tests. The ingestion process is a requirement that unlocks the rest of the features, and we are constantly pushing for improvements.
At Meta, Bento is our internal Jupyter notebooks platform that is leveraged by many internal users. Notebooks are also being used widely for creating reports and workflows (for example, performing data ETL) that need to be repeated at certain intervals.
✏ Data
The one thing that this job has taught me is that truth is stranger than fiction.
Data quality issues have been a looming threat for any and all enterprises, often surfaced by the proliferation of new data analytics and AI projects that, incidentally, rely on good data to succeed.
To be able to model your teams data properly, you need to be able to conceptualize relevant business entities and organize them in a way that is conducive to common questions asked within your organization.
🤖 AI┆ML┆Data Science
You know, Burke, I don’t know which species is worse.
…I wanted to share some of the books, articles, and podcasts that have helped shape my perspective over the past year. This list offers a range of viewpoints on the threats, opportunities, and challenges posed by AI and some thoughtful ideas on how to respond.
After 30 years teaching law, the internet policy legend is as worried as you’d think about AI and TikTok — and he has surprising thoughts about balancing free speech with protecting democracy.
🔥 Catch up
…Next Saturday night, we're sending you back to the future!
🚨 The next section contain my own writing. Don't blame me if you feel distressed after reading this; you chose to read it, although you can skip without thinking twice.
🥷 It will steal 97 seconds from you
Random thoughts, ideas.
The hardest truth I’ve learned as a data engineer is this: No matter how fancy your pipeline or infrastructure is, if your data foundation doesn't have the ability to support the business, everything you do is just 💩.
You put in all your effort to deliver an internal tool to support analytics, but nobody uses it.
Your tool is 💩.
You tune your SQL script to run 2.5x faster, but the data output is “wrong” and leads to “really bad“ decisions.
Your SQL script is 💩.
The lesson here is that anything you do, if you want it to bring value (so that you can lead a meaningful life), make sure it can help your “customer” solve problems.
Put yourself in your customer’s shoes.
Before developing an internal tool, sit down and talk to your DAs and DSs.
When developing a data pipeline, talk to the business to help define “constraints” and “rules” to control the quality and correctness of your data.
So, to apply this lesson and save this newsletter from being 💩…
…I need you…
…yes, you, the “customers” of this newsletter.
I need your feedback on which aspects I need to improve and things that you expect from this newsletter to help me grow as a DE.
(In the comment section or directly contact me through my mail or linkedIn)
I will adjust my work.
Promise. (Unless your ideas is too “wild”)
Switching the context between “your DE work is 💩 if … “ to “I need your feedback“ is… weird."
“Hasta la vista, baby”
-T800, Terminator 2: Judgment Day (1991)
Thanks for scrolling this far ! Convenient subscribe box here in case you want to scroll my newsletter right in your mailbox :D
