🧠 Second Brain
Search
Functional Data Engineering
# What is Functional Data Engineering
Functional data engineering is crucial for data engineers, bringing clarity with “pure” functions and removing side effects. They can be written, tested, reasoned about, and debugged in isolation without understanding the external context or history of events surrounding their execution. ^762878
Its Functional Programming applied to the field of data engineering initiated by Maxime Beauchemin with Functional Data Engineering — a modern paradigm for batch data processing.
# Why Functional Data Engineering
As ETL pipelines grow in complexity and data teams grow in numbers, using methodologies that provide clarity isn’t a luxury; it’s necessary.
It all concerns idempotency to restart a function and recover from a state of failure.
RW Functional Data Engineering - A Blueprint shows an excellent overview of the critical components of data modeling and functional data engineering, which will tie into data products/data contracts more and more, and the fundamental challenges with handling changes and late arrivals compiled into Data Engineering Design Patterns by Ananth Packkildurai.
Origin:
References: Functional Programming, declarative
Created 2022-06-01