Git for Data

Git like versioning on top of object storage/Data Lakes and databases.

See my article on Branch, Test, Deploy: A Git-Inspired Approach for Data and Part 2, Git for Data Applied: Comparing Git-like Tools That Separate Metadata from Data.

Tools to build git for data, either on object storage with plain files or inside a database.

• LakeFS
  • Version control principles applied to data: Branching, merging, and rollback strategies
  • Implementation options: LakeFS, Quilt, DVC, or native open table format capabilities
  • Impact on data engineering workflows: How version control transforms development and testing
  • Data Version Control · DVC (only for ML) -> Acquired by LakeFS (Data Engineering Acquisitions)
• Bauplan (uses Nessie, at least according to their Data Engineering Whitepapers)
  • The programmable Data Lake. Code-native platform for versioned pipelines on object storage with zero infrastructure management. Simple for developers, robust for systems. The data layer for AI systems.
  • Pythonic-based
• Nessie by Dremio
• Lix: An embeddable change control system that enables Git-like features for any file format.
• Lance: Branching and Shallow Cloning in Lance: Towards a “Git for AI Data”
• Apache Iceberg itself has a branching feature: Branching and Tagging - Apache Iceberg™
• OmniGraph: Company Brain

• Dolt: Dolt is a SQL database that you can fork, clone, branch, merge, push and pull just like a Git repository.
• zero-copy / snapshot approaches: modern data warehouses (Snowflake’s zero-copy cloning, BigQuery snapshots) support this pattern.
• Tigris: Fork Buckets Like You Fork Code | Tigris Object Storage
• Supabase: The core approach is full instance branching. Each branch is a completely isolated Postgres database with the entire Supabase stack (Auth, Storage, Realtime, Edge Functions).
• Tigris: The core approach is immutable object versioning with instant bucket forking powered by FoundationDB.
• Guepard: Guepard Instant Database Branching
• Turso: Supports Branching

• GitHub - koordinates/kart: Distributed version-control for geospatial and tabular data

• xet-core by Hugging Face: Xet is on the Hub
• SQLMesh with Virtual Data Environments (great share by Denis Arnaud)

Comment by Antonio Boffa on LinkedIn:

The spectrum from full copies to metadata/catalog-based branching is very clear. It instantly reminded me of “Git is for Data” (CIDR'23 Data Engineering Whitepapers).

I’ve been looking at a related angle on the storage side in “On the compressibility of large-scale source code datasets” (Information Systems ‘25 https://www.sciencedirect.com/science/article/pii/S0164121225000974), where we use a Git-inspired Permute-Partition-Compress strategy to squeeze huge, versioned source-code collections (Software Heritage) while keeping them manageable. Check more on RW On the compressibility of large-scale source code datasets.

Data Engineering Whitepapers shows how Bauplan does it with Nessie:

Source is the DE Whitepaper above from Bauplan on Reproducible data science over data lakes.

Ciro Greco, founder of Bauplan also says:

Engineers care about two things in production:

• They want downstream consumers never to see corrupted or inconsistent data
• And they want to avoid debugging failures under pressure.

His solution is:

To achieve this, a data system must prevent jobs from writing directly to production tables. Every run needs to execute on an isolated branch where all transformations, schema changes, and ingestions occur. If a job fails, the branch must remain open, with the exact failing state preserved, so the issue can be inspected without touching production. See comment on LI

See DE Whitepapers for git for data:

• So You Want Git for Data? | DoltHub Blog
• Fork Buckets Like You Fork Code | Tigris Object Storage

Origin: LakeFS
References: The State of DevOps in Data Engineering, Open Table Formats, git