Data Assets are the result of the declarative pipeline in Dagster, as detailed in Software-Defined Asset. The power of this approach is discussed in Dagster, highlighting how the declarative model is shaping the future of data.
In this model, triggers are associated with assets rather than jobs, simplifying understanding and management. For instance, rather than triggering daily updates based on pipeline completion, we focus on the asset itself, like updating the revenue data each day. Similarly, triggers can be based on the state of an asset, such as re-materializing an asset when its upstream counterpart changes.
The concept of Data Assets encompasses both the definition and the materialization, as defined in Dagster’s documentation. These assets are dynamic and can also be synonymous with Data Product within the Data Mesh framework.
- recommendations are also a form of product
# Asset-based Data Orchestration
For an illustrative example, see Asset-Based Data Orchestration (from DATA + AI Summit 2023) - YouTube, and also refer to Software-Defined Asset.
For a visual representation, refer to Asset-Based Data Orchestration