5 Things All Great Data Engineers Do
Data engineering is like most things. It’s easy to pick up the basics. But it takes a long time to master. More importantly, it takes the right mindset… writes Zahid Hanif, Technical Director at Optima Connect.
I’ve worked with all kinds of data engineers in my 20+ year career. Some very bad. Most pretty good. And a relatively small number I would call truly exceptional.
And it’s this last category — the truly exceptional data engineers — that I’m interested in.
What is it about the best data engineers that sets them apart? Do they have certain traits and habits that separate them from those who are simply ‘good’?
Well, yes. They do.
The 5 Qualities All Great Data Engineers Have:
- They consistently apply database design, performance and engineering best practices
- They possess exceptional lateral thinking skills
- They anticipate future problems
- They balance defensive and pragmatic coding
- They have a growth mindset
1. They consistently apply database design, performance and engineering best practices
To be a great data engineer, it’s about more than just knowing best practices — it’s about applying them, day-in, day-out.
Anyone can be perfect on a good day. But every day, for your entire career? That requires discipline and for good practices to be so deeply internalised that they become second nature.
The very best data engineers also design for failure. That means that they don’t allow their solutions to fail in an intermediate state. The processes in their solutions either succeed or roll back to a consistent state.
2. They possess exceptional lateral thinking skills
The best data engineers always view a problem from multiple angles before choosing the best solution. This requires a combination of mental agility and experience. Experience will, of course, be gained over time. The ability to think laterally, however, is the more important determining factor of whether someone can become a world-class data engineer.
That’s why our interview process for graduate data engineers includes a thinking styles test.
Sadly, only around one-in-ten applicants passes our test.
Why make it so hard? Well, that’s the only way we can be certain that every single one of our data engineers has the potential to become one of the best in the industry.
3. They anticipate future problems
Excellent data engineers plan ahead and always think about scale. That means understanding how their solution will be used, and anticipating problems that might occur.
Take the example of building a data pipeline. The best data engineers aren’t just looking at today’s volume of data, they’re also thinking about the volume it might gather over time, the sensitivity of the data… the retention. How does this solution scale for 100, 1000, 100k, 100m records.
The best engineers will anticipate all of these factors and design their solution appropriately.
4. They balance defensive and pragmatic coding
Trying to code for every conceivable problem before it arises will often lead to over-engineered solutions.
The best data engineers also understand when a future problem is likely enough to cause a problem to be worth pre-emptively coding for.
This results in solutions that are built quickly, elegantly, and yet still provide sufficient defences to a range of the most common issues.
5. They have a growth mindset
In the best data engineering teams you’ll find a collaborative environment with a healthy dose of friendly competition. Central to this is having a growth mindset.
In data engineering, a growth mindset means knowing that there may be other ways to solve a problem that you may not have considered.
It means being happy to take feedback on board — seeing it as a learning opportunity, not as criticism.
It means keeping up-to-date with the latest data technologies. In an industry where tools and technologies are always changing, it’s critical that data engineers don’t make assumptions or simply rely on yesterday’s technology.
And it means not making the same mistakes twice.
It takes each individual to foster their own growth mindset. But much of it comes down to the culture you create for your engineers.
Final Thoughts
As you might have guessed, this is a topic that’s close to my heart. Not just from a personal development perspective. I’m part of the senior management team at Optima Connect, and data engineering is the biggest department we have. From our newest graduate to our most experienced technical architect, we work hard to be the best in the industry.
Spotting and nurturing the five qualities I’ve listed here is one of the ways we do that. But I’d love to hear what other data engineers would include on the list.
