🧠 Second Brain


Search IconIcon to open search

Functional Data Engineering

Last updated Feb 16, 2024

# What is Functional Data Engineering

Functional data engineering is crucial for data engineers, bringing clarity with “pure” functions and removing side effects. They can be written, tested, reasoned about, and debugged in isolation without understanding the external context or history of events surrounding their execution. ^762878

Its Functional Programming applied to the field of data engineering initiated by Maxime Beauchemin with   Functional Data Engineering — a modern paradigm for batch data processing.

# Why Functional Data Engineering

As ETL pipelines grow in complexity and data teams grow in numbers, using methodologies that provide clarity isn’t a luxury; it’s necessary.

It all concerns idempotency to restart a function and recover from a state of failure.

RW Functional Data Engineering - A Blueprint shows an excellent overview of the critical components of data modeling and functional data engineering, which will tie into data products/data contracts more and more, and the fundamental challenges with handling changes and late arrivals compiled into Data Engineering Design Patterns by Ananth Packkildurai.

References: Functional Programming, declarative
Created 2022-06-01