Imperative (Pipelines)

Last updated Feb 9, 2024

An imperative pipeline explicitly outlines how to execute each step in a sequential, procedural manner. In contrast, a declarative data pipelines approach doesn’t specify execution order. Instead, it allows each step or task to determine the most optimal time and method for execution.

The operational specifics (how) should be managed by the underlying tool, framework, or platform. For instance, it might involve updating an asset when upstream data changes. Although both imperative and declarative approaches ultimately produce the same output, the declarative method offers distinct advantages. It benefits from leveraging compile-time query planners and utilizing runtime statistics. This approach enables more efficient computation and pattern identification, potentially reducing the volume of data that needs transformation.

For an in-depth exploration, read more about this in Data Orchestration Trends: The Shift From Data Pipelines to Data Products.

Origin:
References:
Created 2023-12-06