Data Engineering Architecture

Last updated Sep 13, 2023

A good data engineering architecture is hard.

But the best one is the overview from Emerging Architectures for Modern Data Infrastructure - a16z (or Zotero: The Guide to Modern Data Architecture | Future). Check also Perfect DWH architecture.

• Examples and Types of Data Architecture:
  • Data Warehouse
  • Cloud Data Warehouses
  • Data Marts
  • Data Lake
  • Modern Data Stack vs Open Data Stack
  • Lambda Architecture vs Kappa Architecture, Medallion Architecture
    • MapReduce vs Hadoop
  • Metrics Layer vs Semantic Warehouse vs Data Virtualization
    • Metrics, Key Performance Indicator (KPI)
    • Push-Downs vs rollup
    • Data Modeling vs Dimensional Modeling
    • Data Contract
  • Delta Lake architecture where they unified batch processing and Streaming
  • Architecture for IoT
  • Data Mesh
  • Data architectures have countless other variations
  • Data Fabric, data hub, scaled architecture, metadata-first architecture, event-driven architecture, live data stack, and many more. Reis and Housley - Fundamentals of Data Engineering.pdf

Many overlap and more I have listed in Data Modeling.

Updated:

The Modern Data Graph - By Stephen Bailey

See a collection of many more on Data Engineering Architectures Overview or 4 Data Architectures.pdf.

Origin:
References:
Created 2022-04-05