🧠 Second Brain

Search

Search IconIcon to open search

Data Engineering Concepts

Last updated Jan 13, 2025

There are many data engineering concepts. Most of them are described in this second brain. But here’s a table of the most important in data engineering:

Category Data Engineering Topics
Storage Solutions Data Warehouse, Data Lake, Data Lakehouse, OLAP, OLTP, NoSQL, Storage Layer, Data Lake File Formats, Data Lake Table Format, Apache Arrow, Apache Parquet, DuckDB, Iceberg Table, Lance
Data Processing ELT, ETL, EtLT, Reverse ETL, ELTP, Notebooks, MapReduce, Hadoop, Streaming, batch processing, Data Orchestration, Apache Airflow, Dagster, Kestra, Stream Processing, Backfill, Change Data Capture (CDC)
Data Architecture Modern Data Stack, Open Data Stack, Functional Data Engineering, Software-Defined Asset, Declarative vs Imperative, Data Engineering Lifecycle, Data Lifecycle, Data Engineering Architectures Overview, DAD Stack, NGODS, DataOps, Data Product, Microsoft Fabric
Data Management Data Catalog, Data Contracts, Data Governance, Data Quality, Data Security, Slowly Changing Dimension, Data Modeling, Schema Drift, Logging Pipelines, idempotency, Master Data Management, Data Assets, Database Triggers, Bitemporal Modeling, Bridge Tables
Data Modeling & Warehousing Metrics Layer, Semantic SQL, Semantic Warehouse, Data Virtualization, Data Modeling, Dimensional Modeling, Conceptual, Logical to physical Data Models, Data Vault, Relational Model, Normalization, Denormalization, Granularity, Cardinality, Anchor Modeling, Entity-Centric Data Modeling, Focal Modeling, Bus Matrix, One Big Table, Super Table, Fact Table, Dimensions, CTE, Materialized Views
Performance & Optimization Push-downs, Metrics, KPI, CAP Theorem, Full Table Scan, Delta Load
Cloud & Big Data Technologies Cloud Data Warehouses, Cloud Data Provider, Data Fabric, Data Mesh, MLOps, MS OneLake, Cube, Cube Store
Data Integration & Federation Data Orchestrators, Data Lineage, Graph Databases, GraphQL, Analytics API, Data Integration, Data Model Engine, CDC, Datafusion, Ballista

See also Data Engineering Approaches or Data Engineering Vault.


Origin:
References: #publish
Created 2023-08-21