Data Engineering Architecture
A good data engineering architecture is hard.
But the best one IMO is the overview from A15z on Emerging Architectures for Modern Data Infrastructure (int. Link or Zotero).
# Common Architectures
- Examples and Types of Data Architecture:
- Data Warehouse
- Cloud Data Warehouses
- Data Marts
- Data Lake
- Modern Data Stack vs Open Data Stack
- Lambda Architecture vs Kappa Architecture, Medallion Architecture
- Metrics Layer vs Semantic Warehouse vs Data Virtualization
- Metrics, Key Performance Indicator (KPI)
- Push-Downs vs rollup
- Data Modeling vs Dimensional Modeling
- Data Contracts
- Delta Lake architecture where they unified batch processing and Streaming
- Architecture for IoT
- Data Mesh
- Data Fabric, data hub, scaled architecture, metadata-first architecture, event-driven architecture, live data stack, and many more.
- Data architectures have countless other variations
Some overlaps with Data Modeling where I analyze data architecture patterns, data modeling techniques and approaches.
# Data Architecture Images
Updated:
by
Emerging Architectures for Modern Data Infrastructure | Andreessen Horowitz
# People (on top of the illustration)
The Modern Data Graph - by Stephen Bailey (RW)
See a collection of many more on Data Engineering Architectures Overview or 4 Data Architectures.pdf, or Perfect DWH architecture.
More on Data Engineering Architectures Overview and its history.
# Further Reading
- Perfect DWH architecture / Data Engineering Architectures Overview
- Classical Architecture of Data Warehouse
- Deciphering Data Architectures (James Serra)
- Fundamentals of Data Engineering (Joe Reis, Matt Housley) ( Zotero)