Data Engineering

Last updated Aug 21, 2023

• Dataengineering

Data engineering is a term that has shifted over the years from a Database Admins (DBA), ETL Developer, and Business Intelligence Specialist and merged with Software Engineers to a Data Engineer with the growth of data made his title.

It’s still not well defined, the latest book on Fundamentals of Data Engineering (Joe Reis, Matt Housley) tries and does probably best as of today; it’s getting clearer. Besides several boot camps, universities are also starting to get a degree in data engineering like Data Science did before. Let’s start by defining what data engineering is.

Data engineering is the less famous sibling of data science. Data science is growing like no tomorrow, as does data engineering, but much less heard. Compared to existing roles, it would be a software engineering plus business intelligence engineer including big data abilities as the Hadoop ecosystem, streaming, and computation at scale.

Business creates more reporting artifacts, but with more data that needs to be collected, cleaned, and updated near real-time, complexity is expanding daily. With that said, more programmatic skills are required, similar to software engineering. The emerging language at the moment is Python (more The Tool Language, Python) while used in engineering with tools identical to Apache Airflow, Dagster, other Data Orchestrators, and data science with powerful libraries. Today as a BI engineer, you use SQL for almost everything except when using external data from an FTP server, for example. You would use bash and PowerShell in the nightly batch jobs. But this is no longer sufficient, and because it gets a full-time job to develop and maintain all these requirements and rules (called pipelines), data engineering is needed.

More on What is Data Engineering .

• The State of Data Engineering
• Data Engineering, the future of Data Warehousing? | ssp.sh
• Business Intelligence meets Data Engineering with Emerging Technologies | ssp.sh
• The Evolution of The Data Engineer: A Look at The Past, Present & Future

# Data Engineering Best Practices

• Data Engineering Concepts
• Data Engineering Guides

Links:

• Data Engineering, the future of Data Warehousing? |
• data-engineering-explained-in-short
  • Simon Spaeti’s answer to What is data engineering? What does a data engineer do? What are the typical responsibilities of a data engineer? - Quora
  • How to become a data engineer - Quora
• Data & Data Engineering — the past, present, and future | by Zach Wilson | Medium
  - How to learn data engineering
  • The Modern Data Stack: Past, Present, and Future
  • Emerging Architectures for Modern Data Infrastructure - a16z
• Introduction to Data Engineering - hadoop
• Fundamentals of Data Engineering (Joe Reis, Matt Housley)
• Data Glossary 🧠
• Data Engineering Wiki - Reddit
• Practical data engineering concepts and skills
• How to learn data engineering
• Best of Zach Wilson
• The Evolution of The Data Engineer: A Look at The Past, Present & Future

If you are new to data engineering you should start by reading the holy trinity from Maxime Beauchemin. He wrote some years ago 3 articles defining the data engineering field.

• The Rise of the Data Engineer
• The Downfall of the Data Engineer
• Functional Data Engineering — a modern paradigm for batch data processing

Books:

• Data Pipelines Pocket Reference [Book]

Origin:
References: SQL The role of a data engineer Data Engineering When is a data engineer needed Data engineer job description and skills
Created: 2021-10-11