Declarative vs. Imperative

The difference between declarative and imperative is the describing of how something is done. An imperative data pipeline, for example, needs to implement all the technical details of uploading to S3, implementing against the API.

Whereas a Declarative Pipeline only declares that it will upload to S3, but not how. That is left to the “system”. Think of it as Kubernetes or data-aware orchestration with Software-Defined Asset.

The comparison in one picture shows it better:

Declarative vs. Imperative Overview, Initial source from Use declarative pipelining instead of imperative by Ascend.io, and also Data Orchestration Trends: The Shift From Data Pipelines to Data Products

Here’s an example of Ascend.io a while back, when you kick off a declarative pipeline, where you define your partitions, your upstream dependencies, and data in and output.

The system then checks everything, you as the data pipeline author don’t need to implement it.

The full architecture back then with its control plane architecture, that has components which take care of the how:

Gremlin is the graph traversal language of Apache TinkerPop, and they have a great example on their website about Gremlin traversals and how they can be crafted in imperative ( procedural), declarative ( descriptive), or a combination of both styles.

Imperative Gremlin Traversals dictate the exact steps for traversers. For example, an imperative traversal might start with a traverser at the Gremlin vertex, then branch out to Gremlin’s non-Gremlin collaborators, and finally to their managers, culminating in a distribution of manager names. This method explicitly directs the traversers, outlining a clear, procedural path.

Short example:

1
2
3
4
5

g.V().has("name","gremlin").as("a").  
  out("created").in("created").  
  where(neq("a")).  
  in("manages").  
  groupCount().by("name")

Declarative Gremlin Traversals, in contrast, do not specify the traversal order. Instead, they allow traversers to choose from a set of (potentially nested) patterns. A declarative traversal, while achieving the same outcome as its imperative counterpart, benefits from both compile-time and runtime query planning. The latter selects the most efficient traversal pattern based on historical data patterns, prioritizing those that most effectively reduce or filter data.

1
2
3
4
5
6
7
8

g.V().match(  
  as("a").has("name","gremlin"),  
  as("a").out("created").as("b"),  
  as("b").in("created").as("c"),  
  as("c").in("manages").as("d"),  
   where("a",neq("c"))).  
  select("d").  
  groupCount().by("name")

More details can be found in TinkerPop’s Gremlin documentation.

Further reading includes Ascend’s insights:

• Use declarative pipelining instead of imperative
• Declarative Pipeline
• Declarative Pipeline Workflows & DataOps Challenges
• Declarative Pipelines to DataOps Workflows.
• The State of DevOps in Data Engineering

This concept is closely linked with DevOps, where tools like Kubernetes and Descriptive Configs - YAML have revolutionized deployment practices. A similar transformation is needed in data pipelines to enhance stability, accelerate development cycles, and ensure scalability.

Origin: Ascend.io
References: Dagster
Created: