Data Assets

Last updated Feb 15, 2024

Data Assets are the result of the declarative pipeline in Dagster, as detailed in Software-Defined Asset. The power of this approach is discussed in Dagster, highlighting how the declarative model is shaping the future of data.

In this model, triggers are associated with assets rather than jobs, simplifying understanding and management. For instance, rather than triggering daily updates based on pipeline completion, we focus on the asset itself, like updating the revenue data each day. Similarly, triggers can be based on the state of an asset, such as re-materializing an asset when its upstream counterpart changes.

The concept of Data Assets encompasses both the definition and the materialization, as defined in Dagster’s documentation. These assets are dynamic and can also be synonymous with Data Product within the Data Mesh framework.

The importance of Data Product is exemplified by DJ Patil, emphasizing that

• recommendations are also a form of product

Ascend.io pioneered this approach over a year ago, as seen in Use declarative pipelining instead of imperative and Declarative vs Imperative.

For an illustrative example, see Asset-Based Data Orchestration (from DATA + AI Summit 2023) - YouTube, and also refer to Software-Defined Asset.

For a visual representation, refer to Asset-Based Data Orchestration

Origin: Introducing Software-Defined Assets | Dagster Blog, Rethinking Orchestration as Reconciliation: Software Defined Assets in Dagster | Elementl - YouTube
References:
Created: 2022-04-28