DataOps

Last updated Feb 9, 2024

DataOps, in my opinion, is synonymous with the Data Engineering Lifecycle. It represents a culture that integrates and manages the lifecycle in a lean and agile manner, inspired by DevOps and Product thinking.

Sourced from: The Rise of DataOps. Have we found a fix for today’s data… | by Prukalpa | Sep, 2022 | Towards Data Science

Insightful readings:

• RW Why DataOps Is Here to Stay
• RW The Rise of DataOps
• RW Data Engineering Declarative Pipelines to DataOps Workflows.

Sources: Unraveldata

Both MLOps and DataOps share several aspects:

• Collaborative workflow: Both embrace a philosophy of harmony and speed through cross-departmental collaboration.
• Automation: They aim to automate processes in their respective pipelines, from data preparation to reporting in DataOps, and from model creation to deployment and monitoring in MLOps.
• Standardization: DataOps standardizes data pipelines, while MLOps standardizes ML workflows, establishing a common language for stakeholders.

Key Differences:

• They address distinct questions and goals in the machine learning lifecycle, requiring unique expertise and tools.
• DataOps can exist independently of MLOps, focusing on data extraction and transformation, while MLOps inherently relies on data operations.
• DataOps applies across the entire data application lifecycle, whereas MLOps focuses on managing and deploying machine learning models.
• The primary goal of DataOps is to streamline data management, accelerate market delivery, and ensure high-quality outputs. In contrast, MLOps centers on facilitating ML model deployment in production environments.
• Source

Exploring the lifecycle of DevOps:

Also related GitOps.

Origin:
References:
Created 2022-05-22