Data Engineering Architecture

A good data engineering architecture is hard.

But the best one IMO is the overview from A15z on Emerging Architectures for Modern Data Infrastructure (int. Link or Zotero).

• Examples and Types of Data Architecture:
  • Data Warehouse
  • Cloud Data Warehouses
  • Data Marts
  • Data Lake
  • Modern Data Stack vs Open Data Stack
  • Lambda Architecture vs Kappa Architecture, Medallion Architecture
    • MapReduce vs Hadoop
  • Metrics Layer vs Semantic Warehouse vs Data Virtualization
    • Metrics, Key Performance Indicator (KPI)
    • Push-Downs vs rollup
    • Data Modeling vs Dimensional Modeling
    • Data Contracts
  • Delta Lake architecture where they unified batch processing and Streaming
  • Architecture for IoT
  • Data Mesh
  • Data Fabric, data hub, scaled architecture, metadata-first architecture, event-driven architecture, live data stack, and many more.
  • Data architectures have countless other variations

Some overlaps with Data Modeling where I analyze data architecture patterns, data modeling techniques and approaches.

Updated:

by Emerging Architectures for Modern Data Infrastructure | Andreessen Horowitz

The Modern Data Graph - by Stephen Bailey (RW)

See a collection of many more on Data Engineering Architectures Overview or 4 Data Architectures.pdf, or Perfect DWH architecture.

• Perfect DWH architecture / Data Engineering Architectures Overview
• Classical Architecture of Data Warehouse
• Deciphering Data Architectures (James Serra)
• Fundamentals of Data Engineering (Joe Reis, Matt Housley) ( Zotero)

Origin: Data Engineering Architectures Overview
References:
Created 2022-04-05