The core question is: “When should you store data internally in your warehouse versus externally in object storage?”. Hot data queried frequently goes inside. Cold archival data stays external, where it’s cheaper but slower. Interestingly, Databricks and BigQuery recently added external table features, but why? Not because they’re trendy, but because the economics still work.

This article offers an inside look at external tables, their 25-year history, how they evolved from CSV parsers to ACID lakehouse tables, and whether you need to know about them today.

What Are External Tables?

So what are external tables, and why have we been using them for so long? Why don’t we just use the internal storage of a database?

In Oracle, where I first used them in 2008, they allowed you — and still do — to access data in external tables. External tables are defined as tables that do not reside in the database, and can be in any format for which an access driver¹ is provided. All of this is provided via DDL (Data Definition Language) of the database, describing an external table with all its columns, data types, etc., exposing the data as if it were residing in a regular database table.

The external data can be queried in parallel and queried directly using SQL. Essentially, it’s read-only access to data stored outside of our database, making it available in a tabular, easy-to-work-with format to interact with existing tooling and language. In 2008, this was through procedural language such as PL-SQL in Oracle or T-SQL on MSSQL.

Today, external tables have evolved. The biggest change is that they can read more formats including semi-structured data such as Parquet, JSON, Avro, and ORC. While CSV was readable in 2008, the difference today is the columnar formats and nested formats that enable faster analytics. These are available for downstream processes and dashboards, but mostly accessed through SQL queries in one form or another.

A modern definition by BigLake, an evolution of BigQuery toward a multi-cloud lakehouse that tries to solve key customer requirements around the unification of data lake and enterprise data warehousing workloads, introducing external tables in 2015 as part of it²:

External tables are stored outside of BigQuery storage and refer to data that’s stored outside of BigQuery. [..] Google Non-BigLake external tables let you query structured data in external data stores. To query a non-BigLake external table, you must have permissions to both the external table and the external data source.

Snowflake defines them as:

[…] When queried, an external table reads data from a set of one or more files in a specified external stage, and then outputs the data in a single VARIANT column. Additional columns can be defined, with each column definition consisting of a name, data type, and optionally whether the column requires a value (NOT NULL) or has any referential integrity constraints.

External tables were added in 2021, and Snowflake described their benefits as follows:

External Tables Address Key Data Lake Challenges:

1. To augment an existing data lake. [..] augment their existing data lake, rather than replace it. The External Tables feature enables that use case. Customers can use external tables to query the data in their data lake without ingesting it into Snowflake. (side note: MVs³)
2. Ad-hoc analytics. Customers often use external tables to run ad-hoc queries directly on raw data before ingesting the data into Snowflake. Ad-hoc queries help them evaluate data sets and determine further actions.

Just a Pointer (Symlink)?

A simple analogy is a symlink in Linux, where you point from your current directory to another directory without moving data. You just add a pointer. If you read that file from that symlink, all it does is read it from the location the symlink points to.

An external table is the same, just a pointer to external data, bringing that data into the current data warehouse or cloud solution, hence the word external. You define the source format such as XML, CSV, etc., and define their structure, and then you can query that at any time. It’s similar to a SQL View in that sense, but pointing to non-internal data.

Running DROP TABLE and deleting an external table is metadata-based only. No data is removed, only the table definition from the internal data catalog. The same is true with a symlink. Almost any relational database today has support for it, even if it’s not called an external table. Everyone occasionally needs to read data outside of its warehouse or database.

Recap in the History of External Tables

Looking back at the history and evolution of external tables, we can quickly see that there’s a long history and they’ve been a recurring pattern across every generation of database technology since the early 2000s, and arguably longer if you count IBM’s federated database concepts from the late 1990s.

The Origin Story: ISO in 2001

The history starts with ISO/IEC 9075-9, published in 2001. Part 9 of the SQL standard defined foreign-data wrappers and datalink types for managing external data from within SQL. The work was completed in late 2000 and published alongside SQL:1999, with full integration in SQL:2003 (it was later updated in 2023).

It was the initial definition and extensions to database language SQL to support management of external data through the use of foreign-data wrappers and datalink types.

My first encounter was with Oracle external tables, but according to Wikipedia there were earlier implementations, such as Microsoft Access linked tables (~1992). Microsoft Access linked tables (~1992) were the earliest consumer-facing implementation where users could link dBASE, Paradox, text files, and ODBC sources as if they were Access tables. ODBC 1.0 (1992) itself established the first standard for heterogeneous data access across databases, though it didn’t create table abstractions.

Further, IBM’s DB2 DataJoiner (~1995) was more ambitious with a middleware product enabling SQL queries across Oracle, Sybase, SQL Server, Informix, Teradata, and even VSAM files through a unified interface. With SQL Server 7.0’s Linked Servers (1998) we got federated querying to Microsoft’s ecosystem via OLE DB, supporting cross-database joins with four-part naming conventions.

Most of these implementations shared a common limitation that Oracle (9i Release 1 - 9.0.1 in 2001) solved: they focused on querying other databases or required middleware. Oracle’s abstraction treated local flat files as first-class read-only table objects using the familiar CREATE TABLE ... ORGANIZATION EXTERNAL DDL syntax, providing a simple way to define external files as part of normal table creation and allowing ORACLE_LOADER access to query flat files (CSV, fixed-width, delimited) through DBAs.

It was an early way of separating declaration from compute (the Oracle loaders).

Why External Tables? What Are Their Benefits?

But why use external tables? What makes them so useful that they persisted? Why have they survived so long, and why are they getting added to Databricks and other major platforms?

For that, we need to look at external tables’ benefits. The first reason is that external tables can simplify data access to avoid developing ETL pipelines, moving data out of the source, and re-ingesting it in our data warehouse. They make external data accessible easily, defined in a tabular form by a database schema with column types. Typical cloud data warehouses like Snowflake and Azure use them to link existing data from object storage easily without moving data. This makes the object storage files accessible for almost any downstream tool or query language in a simple and cost-effective way.

Other ways of using them are to store some data on cheaper storage (e.g., object storage over data warehouse storage) and only link them in. It’s slower to fetch, but more affordable to keep. If you have large data sets, cost savings can be immense as this article shows, bringing down Snowflake internal storage cost from ~$23/TB/month to S3 infrequent access with ~$12.50/TB or S3 Glacier Deep Archive with only ~$1/TB.

Another handy side effect as the consumer of external table data is that the data is always up to date, because no refresh or update is needed. It goes without saying that this has its own downsides and can be a problem for the owner of the data if it’s used in production and the ETL process reads large amounts of data through external tables. This will affect upstream apps running or owning this data.

That’s why many use external tables in combination with materialized views (MVs) to truncate and recreate a daily snapshot (or similar) during off-peak (mostly nights) of this data, avoiding affecting production data and even optimizing query performance with added indices for downstream queries.

When Internal and When External Data? What’s the Limit of External?

The tradeoffs come down to how often the data is queried, e.g. the hot versus cold question.

The tradeoffs and considerations you should make when wanting to use them come down to the decision of how often the data is queried. The table below shows it in more detail:

In the era of data lake and lakehouse architectures, this is an important consideration. VSCO says: “disk space was growing more quickly than our compute needs,” which is what triggered the adoption of external tables.

If you look at your use case, if you need to do analytics across various sources with joins and augmentation of your data at an enterprise, you probably want to focus on loading data into your database or data warehouse, an architectural pattern that has survived more than 30 years. But if you have data that is external and small but you want to join it with existing data, or you always need fresh data and can live with a slower response time (maybe because it runs during the night), you might use external tables.

In any case, external tables are a good approach to keep in mind and a valuable toolkit to have.

They Work Well with Existing Tech and Common Patterns

Obviously, today’s external tables are not the same as the earliest ones in Microsoft Access, but the principle of accessing data outside your system is still the same. Nowadays we have more support, new formats besides CSV and JSON. We can do Parquet or open table formats.

As mentioned, they work well with related long-lasting data warehouse patterns and applications such as materialized views and stored procedures. The recurring pattern is to access external data with your data management system, similar to the pattern of materialized views that refresh complex SQL statements and make them fast, and stored procedures that run glue code within your database.

Moreover, there are temporary tables that are similar but only available during a transaction or session. They all work in the same Lindy effect, e.g., Databricks just announced Temporary table support recently on December 9th, 2025, or Databricks SQL Stored Procedure a little earlier, August 14th, 2025, for reusing existing SQL statements.

Again and again, everything that is old will be new again. Exactly what the Lindy Effect is all about. We can clearly say that the Lindy effect over the last 33 years applies here. The longer something is in place, the more likely it is to be around for at least that long.

[!info] External vs. Temporary Table

In contrast: temp table = session-scoped, writable, fast, invisible to others, auto-dropped. External table = persistent metadata, read-only, infinite size, visible to all, optimized for cost.

A common chain in practice is going from: external table → temp/transient table → permanent managed table.

How a Classical External Table Works

To understand how traditional external tables work, let’s first look at Oracle, which has built an extensive syntax around them and where they still work this way today.

First, we can create a place for external data called DIRECTORIES, which is simply a pointer or alias to a file system location where external files already exist:

1
2

CREATE OR REPLACE DIRECTORY admin_dat_dir
    AS '/flatfiles/data';

This directory can point to local file systems, NFS mounts, or even cloud object storage today (with the ORACLE_BIGDATA driver for S3, OCI, Azure). The DIRECTORIES don’t require moving data, though you could prepare those files via ETL pipelines or third-party tools, or they can be generated directly by applications.

We can now create an external table based on this directory, e.g., log files, bad data that we store externally, JSON files, and make data accessible inside the INFORMATION_SCHEMA and with plain SQL, as if it were internal.

Creating an external table:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

CREATE TABLE admin_ext_employees
                   (employee_id       NUMBER(4), 
                    first_name        VARCHAR2(20),
                    last_name         VARCHAR2(25), 
                    job_id            VARCHAR2(10),
                    manager_id        NUMBER(4),
                    hire_date         DATE,
                    salary            NUMBER(8,2),
                    commission_pct    NUMBER(2,2),
                    department_id     NUMBER(4),
                    email             VARCHAR2(25) 
                   ) 
     ORGANIZATION EXTERNAL 
     ( 
       TYPE ORACLE_LOADER 
       DEFAULT DIRECTORY admin_dat_dir  --notice this dir with above
       ACCESS PARAMETERS 
       ( 
         records delimited by newline 
         badfile admin_bad_dir:'empxt%a_%p.bad' 
         logfile admin_log_dir:'empxt%a_%p.log' 
         fields terminated by ',' 
         missing field values are null 
         ( employee_id, first_name, last_name, job_id, manager_id, 
           hire_date char date_format date mask "dd-mon-yyyy", 
           salary, commission_pct, department_id, email 
         ) 
       ) 
       LOCATION ('empxt1.dat', 'empxt2.dat') 
     ) 
     PARALLEL 
     REJECT LIMIT UNLIMITED; 

The first and most important choice is TYPE, which determines the access driver and what kind of files you can read: ORACLE_LOADER for plain text files like CSV or logs (read-only), ORACLE_DATAPUMP for Oracle binary dump files, ORACLE_BIGDATA for cloud object stores like S3 or OCI in formats like Parquet or Avro, and ORACLE_HIVE for Hadoop/Hive data. The DEFAULT DIRECTORY points to a server-side path alias, and LOCATION names the actual file(s), with wildcard support (*.dat) so you can load a whole batch at once.

The ACCESS PARAMETERS block is where you control parsing: row and field delimiters, null handling, custom date format masks, and where to write bad rows (badfile) and parse logs (logfile). On top of that, PARALLEL lets Oracle split file reading across multiple processes for large files, and REJECT LIMIT controls fault tolerance. Set it to UNLIMITED to skip bad rows silently, or 0 to fail immediately on the first error.

You see lots of built-in features that we can use compared to building a full-fledged data pipeline. Instead of exporting and importing CSVs from the source databases or developing a complex CDC pipeline that traditionally looked something like: source OLTP --> CSVs --> IDW (reports on yesterday) -> ingest into DWH for long-term analytics, we can just define a table based on external data and access it as part of our pipeline.

[!tip] The INFORMATION_SCHEMA analogy

You are probably familiar with the INFORMATION_SCHEMA of a database. It’s the internal data catalog that most databases provide and it contains a list of all tables and all metadata such as columns, data types, etc. The neat thing is that external tables will show up as internal tables once defined.

What’s the Modern Version of External Tables Today?

To preface: the previous Oracle example shows the CREATE EXTERNAL TABLE syntax, and a first-class DDL object in the data catalog. What follows in this chapter is the next evolution, where external tables are not necessarily created with DDL, but in another way, achieving the same outcome of querying data in place without loading it. Let’s see what these are.

Integrated into Warehouses

Most modern warehouses - Snowflake, Redshift Spectrum, BigQuery, Athena, Synapse - come with a simplified version of CREATE EXTERNAL TABLE. Compared to the Oracle example, the schema is usually inferred from the file format (especially Parquet), S3 or another object store is the default backing location, and the parsing ceremony disappears. The pseudo-code looks roughly like this across engines:

1
2
3
4
5
6

-- Pseudo-code: modern external table over Parquet on S3
CREATE EXTERNAL TABLE sales
WITH (
  LOCATION = 's3://my-bucket/sales/',
  FORMAT = 'PARQUET'
);

Object storage like S3, GCS, and Azure Blob has become the first-class citizen for external data. From here, the ecosystem layers on: dbt wraps this in YAML, DuckDB skips the DDL entirely in favor of schema-on-read, and open table formats add transactional guarantees on top.

External Tables with dbt?

On top of this base SQL form, dbt adds a YAML layer and can be used with its own package called dbt-external-tables. It’s one of the most-used dbt packages, though it seems less actively maintained now.

The external table is defined via YAML, and there are lots of options to set, with the most important being external and its location, but also defining columns in different ways such as inference or the meta tag:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

version: 2

sources:
  - name: snowplow
    tables:
      - name: event
        description: >
            This source table is actually a set of files in external storage.
            The dbt-external-tables package provides handy macros for getting
            those files queryable, just in time for modeling.
                            
        external:
          location:         # required: S3 file path, GCS file path, Snowflake stage, Synapse data source
          ...               # database-specific properties of external table
          partitions:       # optional
            - name: collector_date
              data_type: date
              ...           # database-specific properties

        # Specify ALL column names + datatypes.
        # Column order must match for CSVs, column names must match for other formats.
        # Some databases support schema inference.

        columns:
          - name: app_id
            data_type: varchar(255)
            description: "Application ID"
          - name: platform
            data_type: varchar(255)
            description: "Platform"
          ...

        # Use `meta` to pass custom column properties (e.g. alias, expression)
        columns:
          - name: raw_timestamp
            data_type: timestamp
            config:
              meta:
                alias: event_timestamp       # rename the column in the external table
                expression: TO_TIMESTAMP(...) # custom SQL expression instead of default value extraction

This is a nice improvement over the ODBC GUI interface. It’s not exactly an apples-to-apples comparison as dbt itself is not a database, but with its supported destinations such as Redshift (Spectrum), Snowflake, BigQuery, Spark, Synapse, and Azure SQL, you see that it will persist in these destinations, mostly data warehouses.

DuckDB with dbt

If you use dbt, you can also use DuckDB with dbt via dbt-duckdb, which is more up-to-date. But DuckDB is not an external table, right?

Yes, DuckDB doesn’t have CREATE EXTERNAL TABLE syntax yet, mostly because it is an in-memory database, but you can achieve the same functionality through other means. DuckDB can not only be used as a database but also as a zero-copy SQL connector (see all categories at 5 Key Categories). We can just point it to an external source, as shown above with dbt. The difference is that DuckDB is both a database and a compute engine, making ad-hoc reads possible directly without a DDL definition, similar to an external table with Oracle loaders. With dbt, we can nicely declare this in dbt configs.

With DuckDB, you can query “external data” extremely fast over HTTPS or locally in formats such as Parquet, CSV, and many more, so the need for formal external tables is reduced since DuckDB does schema on read.

If you want to define the database schema ahead of time, we’d use external tables to do that and effectively have schema on write (though we don’t write, just define the DDL table structure and data types), which is more of the classical ETL approach.

Here’s an example with external_location to read external data with dbt:

1
2
3
4
5
6

sources:
  - name: external_source
    config:
      external_location: "s3://my-bucket/my-sources/{name}.parquet"
    tables:
      - name: source1

Read more at Fully Local Data Transformation with dbt and DuckDB.

Other options are with database views that are supported in DuckDB with CREATE VIEW over read_parquet(). You can ship a .duckdb file to clients with pre-defined views over S3 data, so clients don’t need to know about the underlying data, Hive partitioning, or even glob patterns — very similar to what a formal CREATE EXTERNAL TABLE would do.

1
2

CREATE VIEW events AS
  SELECT * FROM read_parquet('s3://lake/events/*.parquet', hive_partitioning=true);

Or similarly use ATTACH to directly point to Postgres, MySQL, SQLite, S3, and others:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

-- Postgres (binary wire protocol, predicate + projection pushdown, read+write)
INSTALL postgres; LOAD postgres;
ATTACH 'dbname=postgres user=postgres host=127.0.0.1' AS pg (TYPE postgres);
ATTACH 'postgresql://user@host/db' AS pg (TYPE postgres, READ_ONLY);

-- MySQL (via MariaDB Connector/C; Postgres-style keyvalue string even for MySQL — easy trap)
INSTALL mysql; LOAD mysql;
ATTACH 'host=localhost user=root port=0 database=mysql' AS mdb (TYPE mysql);

-- SQLite (file opens directly; multi-reader single-writer by SQLite file locks)
INSTALL sqlite; LOAD sqlite;
ATTACH 'sakila.db' (TYPE sqlite);

-- Generic remote DuckDB file
ATTACH 's3://duckdb-blobs/databases/stations.duckdb' AS stations_db;

Open Table Formats and Lakehouse Architecture

That begs the question of whether Open Table Formats are the next evolution and modern way of external tables. These table formats allow almost any SQL compute engine to use them as external tables, and read, compute, and aggregate as a database would.

If we look at what table formats consist of, they’re built on object storage, with a file format like Parquet, and then we have a manifest file that contains a list of files that unifies multiple single files into a “single” table, looking from the outside.

So again, the manifest file is our pointer or fancier symlink, but it lives next to the data, unlike external tables. There’s much more going on in table formats, but if we have a data lake with open table format tables, we can see how we define tables in DDL and the pointers are to different files (Parquet, ORC, Avro), in most cases Parquet.

More broadly, we can say external tables decouple storage from compute. Open table formats decouple the table itself (schema, history, transactions, statistics) from any single engine.

Lakehouse and Connecting to DuckLake

One step further is obviously a lakehouse architecture, with the shift from format-agnostic file reading to governed, transactional, multi-engine open table formats.

If you extend the external table idea to a lakehouse architecture, these external tables with open table formats provide essentially what databases provide with ACID guarantees, time travel, schema evolution, partition evolution, and fine-grained access control, but for files.

But with the difference that data stays in open Parquet file format on customer-owned cloud storage. The external table, once a humble workaround for avoiding data loads, has become the architectural foundation of the data lakehouse if you like this analogy.

With DuckLake, we have the next evolution just around the corner, bringing back exactly that missing database, especially to handle all the metadata of such a lakehouse and all its files. This means having durable and consistent database storage for our manifest files.

Open Data Catalog to Complete the Picture: The ODBC Glue

With all these evolutions, we’ve come far. When adding an Open Data Catalog, we are exactly where we started: having an INFORMATION_SCHEMA, a dictionary with all our tables, in this case the open table format tables.

It’s the glue that ODBC provided when connecting a BI tool to the underlying database. Now you’d like to have an open data catalog that, in the best-case scenario, gives you all the tables and ways to connect.

But then again, the syntax of EXTERNAL TABLES still gets added, and ADBC and DuckDB are doing a great job of using external data without needing a data lake and its technology stack altogether. For example, DuckDB has support for ODBC, ADBC and even JDBC. That matters especially for 3rd-party tools: ADBC streams Apache Arrow end-to-end instead of serializing row-by-row, so BI tools and notebooks can pull millions of rows directly from external Parquet tables at speeds that previously required keeping data “hot” in a cloud data warehouse. 🙂

[!note] ADBC, what is that?
ODBC is 30+ years old, and we have a newer, faster version of it, called ADBC. It’s a faster way to connect to other databases with a columnar-oriented API instead of row-by-row serialization, heavily making use of Apache Arrow.

While ADBC is newer, it tries to support the same drivers as ODBC, but faster and easier to install. E.g., it has a handy dbc CLI to install it on almost any programming language, so no more manual and error-prone Windows GUI ODBC downloading of drivers and definitions needed, just one CLI command.

[!tip] Using MotherDuck
If you want a data warehouse that just works, integrates well with DuckDB, and has support for DuckLake, you can always use managed MotherDuck. You can build a classical data warehouse with plain SQL, you can read external data easily with DuckDB or dbt-duckdb, or integrate with DuckLake.

It works great with agents. Check out MotherDuck’s agent-skills for opinionated AI skills for building applications with MotherDuck. And visualize with Dives with one prompt.

Which Is Faster? A Quick Benchmark

To put numbers behind the hot/cold decision, I ran a simple benchmark on the TPC-H SF=1 lineitem table (6M rows, ~150 MB), stored four ways: inside a DuckDB file (internal), as raw Parquet, as an Iceberg table, and as a DuckLake table. Full code: bench2.py and metadata_bench.py.

Dashboard workload (hot path): 3 queries × 10 repeats:

Internal is fastest; external pays a 1.3×–1.7× tax. But for cold/archival queries (one-off, no warmup), all four backends answered in under 150 ms. The speed difference effectively vanishes for data you query once a week.

Storage cost is where external tables shine. Columnar Parquet is ~40% smaller than native DuckDB format. Ten TB of archive data costs roughly ~$125/month on S3 Infrequent Access or ~$10/month on Glacier Deep Archive, versus ~$230/month inside Snowflake on capacity pricing. This is the economic case external tables were invented for, and it still holds.

Metadata workload is where DuckLake stands out. Fifty single-row inserts showed DuckLake creating zero data files (rows inlined in the catalog) versus Iceberg’s 352 files (201 data + 151 metadata). That’s the “small file problem” made concrete: at one write per second, Iceberg creates ~86,400 files per day needing compaction. DuckLake creates zero until you checkpoint. DuckDB Labs’ own benchmarks report up to 926× faster queries on streaming workloads.

So Should You Use External Tables?

So after all this, should you use external tables today? After seeing how sticky they’ve been since Oracle 9i in 2001, how they keep getting re-added to newer tools (Snowflake in 2021, Databricks Unity Catalog, BigLake in 2022), and how their core benefit is. Accessing data where it lives without moving it, via a simple DDL statement, has only grown more valuable as formats have evolved from CSV to Parquet, JSON, Avro, and now open table formats. I’d say yes. But choose wisely based on your data’s temperature: use internal storage for hot data, such as dashboards and frequently used queries.

Use external tables for cold data, archival workloads, and ad-hoc exploration, where that gap vanishes, and storage costs plummet (up to 20× cheaper on Glacier Deep Archive vs. warehouse-managed storage). And if you already use dbt, DuckDB, or a lakehouse stack, the modern versions are right there. Where they’re the wrong choice is the inverse: transactional workloads, queries that need sub-second latency on every run, or data so small that the operational overhead of an external stage outweighs the benefit of not loading it.

The evolution is worth naming explicitly: “read CSVs on disk” → “read Parquet on HDFS” → “read Parquet on S3 via a metastore” → “read Iceberg/Delta tables with ACID on S3” → “the Iceberg table is the warehouse table”. Each step kept the core idea (data stays where it lives, metadata describes it, SQL queries it) and added database semantics back in. With open data catalogs, the warehouse becomes a stateless rental over a bucket you own, and external tables are increasingly managed. DuckLake demonstrates this best: when the catalog has SQL-DB-like guarantees, the distinction between “external” and “internal” dissolves. The metadata benchmark made this concrete by reading a single indexed row rather than walking a manifest tree.

The database semantics are returning with DuckLake, managed Iceberg, and predictive optimization, all of which reintroduce RDBMS-style guarantees to the lake. The cycle from “external table for cheap storage” to “external table as a full ACID database on S3” took 25 years, completing the journey back to database principles while maintaining the separation of storage and compute. You can say the modern external table isn’t external anymore. DuckDB reads them directly, and DuckLake handles the metadata that multifile lakehouse architectures would otherwise drown in. The lesson from history is that whenever someone tries to replace it, the pattern is that reading data in place always beats moving it. And the Lindy Effect suggests that if external tables have lasted 25 years and get re-added, they’ll persist another 25. They’re probably not going anywhere. 🙂

Full article published at MotherDuck.com - written as part of my services

This article looks at how BI evolved, how dashboards are actually used today, and what survives when AI enters the picture — starting with the foundation that was never really about dashboards in the first place, and ending with the problem nobody in the AI hype cycle wants to talk about: who maintains it all.

The Verdict of People in the Field: BI is Dead (Again)

We’ve heard it all. Business intelligence (BI), and especially dashboards, are dead. But every time, only to rediscover its power and resurrection whenever we need grounded data analysis in any enterprise and startup space. The same way Excel never dies, which arguably is still the most used BI tool.

If we look at what others from the data world say, it does sound similar. Hex says that dashboards were never the destination and specifies:

Static reporting surfaces were always a workaround for messy data and limited tooling — agentic analytics finally closes the gap between visibility and genuine insight.

In other words, dashboards were a workaround created because data was messy, tooling was restrictive, and asking open-ended questions of the warehouse wasn’t possible without coding. Hex continues that dashboards create more questions than they answer.

But they also say: respect dashboards, but stop treating them as the goal:

Dashboards still matter. They’re excellent for reporting KPIs, surfacing operational signals, and aligning leaders around shared metrics. They allow data teams to be creative in how they display these metrics and will remain a primary surface for need-to-know numbers.

Dashboards don’t reason or explain why something happens, as the Hex article says. Hex argues that reliable, timely, context-aware answers are the destination.

Mike, CEO of Rill, says:

Agents are blind—they can’t see dashboards. But they do need access to the primitives behind them. AI, and agentic working won’t kill BI, but it will make dimensional modeling, metrics, OLAP cubes, query performance, and governance more important than pretty charts. Agents will reveal what many of us have known for a long time: BI was never about dashboards.

Benn Stancil was already saying in 2021 that BI is dead, where he drew parallels to the Salesforce “End of Software” declaration in 2000. This is very interesting as it’s almost the same statement today with AI, no more software developers needed. Benn argues that the original BI stack was one tool across the stack, but that by 2021, tools were unbundling into dedicated and specialized tools for each layer.

The proposed future of BI should focus only on data consumption for humans, integrating both self-serve applications and deep ad-hoc analysis. This new BI would be “legless” (the opposite of headless, he argued back then), relying on global governance layers rather than proprietary semantic layers, and fostering cross-functional collaboration.

The aim is a universal tool for all data consumption, moving beyond its current diminished state of “visualization and reporting”. Again, very similar to today’s discussion, where everything is about semantics and context. Benn concluded that “BI is dead, long live BI” 🙂.

Take any modern data and analytics discipline, and you’ll probably find it has its roots in the work that has been historically carried out by Business Intelligence developers, the OG jack-of-all-trades of the data industry.

[!note] Find more Opinions and Articles

Other opinions on the world wide web, whether BI is dead. Long live business intelligence., or a discussion on Reddit about Will Business Intelligence skills (BI) be irrelevant in like 3-4 years? : r/analytics or AI kill BI?.

BI Was Never Just About Dashboards

With respect to Benn’s article in 2021, have we come full circle five years later, with the end of software engineering again, and everyone demanding semantics and metrics layers?

At least on the surface, it seems we are at the same point, but today we’re going back to one “Original BI” stack as drawn in the image in the article, to a fully encompassed data platform. Maybe it does not need to be one single platform, but at least to the user it needs to be a single chat or AI interface, that goes end-to-end through all the layers of discovery, visualization, transformation, storage, and ingestion—or in other words, the full data engineering lifecycle.

So what about dashboards then? Many declare the death of many things, and dashboards are a popular target too. That’s even more true when AI, with its generative capabilities, can just one-shot your whole dashboard and create a full-blown web app or custom HTML page. And dashboards might die, as many don’t actually want the dashboard itself, but the extracts from your large SAP, linked to the right customers from the CRM, enhancing the decision-making process even more.

They want the insights from the combination of all source data and business insights that your company has over its competitor, for example. It’s the primitives behind the dashboards that matter more.

When You Still Need a Dashboard (AI Chat is Not Enough)

Even though a chat interface or an agent can provide you with dashboard information tailored to your question and in an explanatory written form, there’s still a need for dashboards in certain situations.

The obvious one is the well-crafted operational dashboard, where you can see your whole company performing in a split second by looking at a single, highly dense dashboard with individual charts and visualizations tailored to convey information about each sub-area in the best possible way. It’s the same way a map is still needed in self-driving cars: for quick verification, to get an overview, or in case the car gets lost.

The other obvious benefit of the operational dashboard, or any general dashboard, is that people can agree on numbers, as they’re looking at the same agreed-upon dashboard. Hence, the calculations are the same compared to individual Excel sheets with different calculations.

The easy creation of multiple dashboards on the fly or chatting with AI to get insights resembles the old way of using local Excel files. Everyone is doing their own thing, with no alignment, governance, or broader verification.

Maps are a different type of dashboard that can’t be replaced. Geospatial data shown on a map still beats text. Or a bar chart where you immediately scan the different proportions of stock in inventory across certain regions, compared to first having to analyze all numbers in a text-only chat reply.

Dashboards as a Sanity Check

Maybe some even less obvious reasons: serendipitous discovery of anomalies and outliers that accidentally pop up in dashboards are harder to see in chats, but easy to spot visually. The same goes for ad-hoc BI when drilling down into more grain and dimensions for self-service users. A pivot table is a REPL for BI; that’s not possible with chats.

Dashboards are also a lifeline to quickly check if the AI is hallucinating. What about determinism? Chat responses are not deterministic, and you might get different responses, hopefully with the same correct answer, but visualized differently because the model made different decisions the second time, or for a different user. AI agents are non-deterministic statistical models and probably always will be, so we need to bring more context and definitions to make outputs more consistent. One way is with a spec-driven development (SDD) first approach that helps define it more accurately each time.

The same applies to stewardship, reviewing and checking outcomes. As an example consider self-driving cars such as Waymos: you don’t need a map, but if it’s wrong or stuck, the first thing you’d reach for is one.

But what does that mean for the future of BI, and what keeps BI afloat?

The Primitives of a Dashboard (and BI)

BI was and is never (only) about dashboards. BI is used for a combination of reasons. One of the most important is the metrics themselves, the business KPIs: defining your profit, defining what has to be deducted, how much taxes and salary payments are. All of it is ingrained in a single metric, or better, all of it including the hierarchy, the full tree of metrics, as it’s never just one metric.

But the metrics tree with all its hierarchy is not all. We need speed to crunch the data extracted from the source SAP and CRM, cleansed and joined, and aggregated to the exact grain of the metric. No one has a single place to view their data end-to-end, which makes BI work so needed, integrating to either combine all sources into one API or database/warehouse for the business (or now AI chat) to pull from. Again, queries over CSV are easy. Fast queries over 1 TB of Parquet are hard.

This is where we need optimal data modeling and compute power to do it in seconds, best case sub-second. And when we have this, we need a good data architecture with the right tool to compress and compute that data. That’s where we need some kind of OLAP or analytical database, optimized for analytical queries and doing aggregation on the fly based on different dimensions and grains.

All of these primitives are still needed, maybe even more so in times of generative AI, even if dashboards were to go away.

Lastly, all of this is under the umbrella of context and semantics. It’s encoding the business-related processes into data-like artefacts to make reusable governed definitions of metrics like revenue, MAU (Monthly Active Users), ROAS (Return on Ad Spend), etc. And the right medium for that is metrics and a so-called metrics layer (or semantic layer).

Everyone Wants to Build Dashboards. Nobody Wants to Maintain Them.

The elephant in the room that nobody talks about is that everyone wants to build. Nobody wants to maintain. This is the current AI phase we’re in. Even more, nobody even wants to review tons of AI-generated code.

According to the story in the book Maintenance of Everything by Stewart Brand, what makes people do maintenance is when they find joy in it and care for it.

This is greatly illustrated in the opening with the 1968 Golden Globe solo sailboat race, a dramatic contest of maintenance styles under life-or-death conditions, contrasting Robin Knox-Johnston’s meticulous upkeep with Donald Crowhurst’s neglect. Knox-Johnston, who truly enjoyed his boat Suhaili, reported mid-race that despite the brutal Southern Ocean ordeal, he was “thoroughly enjoying himself”. Decades later, he personally restored her, replacing every one of her 1,400 fastenings.

Stewart Brand also argues that you need to care about your product, and that’s the key to good maintenance — a lesson drawn from Zen and the Art of Motorcycle Maintenance by Robert Pirsig (1974), a philosophical classic celebrated as bending the culture of the day toward honoring maintenance.

And one way to ease maintenance is to design maintenance-friendly. Like the Rolls-Royce Silver Ghost, which was engineered from the ground up for reliability and ease of upkeep.

So how do we apply these principles to BI? We need to create dashboards that are easy to maintain and have clear ownership, someone who cares. Otherwise, they lose long-term value, and we invest precious time in something short-term.

[!quote] Quote by Mehdi
You just burned $1k in tokens to rebuild the SaaS you refused to pay $29/month for.
Great for the ego. Now maintain it for 3 years to break even.
Yup, maintenance is a luxury these days.

Generating 100s of cool but unused dashboards with AI is clearly working in the opposite direction. We can save a couple of bucks while no longer needing to maintain each custom-created dashboard. Indeed, maintenance is a luxury these days.

Self-Maintaining Agents?

The question is, do we need agents for the maintenance then? So that we can create new, innovative solutions, do the creative things, and leave the hard parts of BI to us humans such as getting requirements and verifying with the business. And we create agents to do the maintenance? What does maintenance even mean? What does changing the oil and checking the brakes mean for BI, or data pipelines?

It’s not troubleshooting in case of an error, that’s what agents already do and help a lot with, ideally fixing the errors themselves, in a self-healing process. But maintenance is different. Keeping up with the updates of your software, security, the right integration into your data platform with upgraded, better-performing glue code, avoiding code from becoming legacy code.

All of it means maintenance, and outweighs the work of just creating the data pipeline or BI dashboard by far, probably somewhere on the order of 8:2.

As Mike puts it well when we were discussing:

Few can build a maintainable, scalable data infrastructure for surfacing trusted metrics. E.g. a digital platform like Coinbase isn’t going to YOLO its internal reporting over billions of transactions. Even Claude has a usage-based billing portal, consumption metrics need to be precise, deterministic, and fast.

The Basecamp owner similarly said: AI is pushing back too little. Maybe it will be solved in the future with better models, but we need to live in the present, and in that present:

Agents don’t finish beautiful, ergonomic, desirable software. They just don’t. That human finishing at the end is not just necessary, it’s essential.

So the future is soon, but not yet. Back to dashboards and their maintenance: the hard part is not generating visualizations, but having metrics and a strong BI backend. Almost a unified data interface that has an agent with access to source, ETL, and dashboard.

BI-as-Code: One Solution to Maintenance?

Is the solution to maintenance-friendly design maybe BI-as-Code?

BI-as-Code comes into play because declarative configs can be versioned and maintained, thereby avoiding the limitations of BI-as-clicks. Sure, it will not solve all problems, but having that descriptive state of A. your data infrastructure and B. your data pipelines and BI dashboards helps tremendously. In the event of an error or incorrect state, we can just roll back to the last versioned dashboard or infrastructure.

The only thing hard to make reversible, unless you use some kind of Git for Data workflow with LakeFS, Nessie, or others, or just use Open Table Formats with the Time Travel function, is the actual data.

BI-as-Code isn’t the whole answer, but it’s the right direction: making dashboards owned, versioned, and recoverable. Code can build the right level of abstractions for ETL, metrics queries (metrics SQL), and visualizations, where raw Python for ETL or D3 is too verbose and too brittle.

With agents, these abstractions come in handy once more, as agents work best with a clear interface like a CLI or API, where the abstraction helps build just that, and tune things themselves through MCP or direct access to the declarative configurations. Much of what GenBI is all about. The question of what comes next: can agents take over the analyst role entirely, or how do we marry the two?

Building BI for Agents, Not Humans

BI-as-Code allows agents to drive BI, or as Mike said: “AI drives compression of the data stack”. Meaning observability, cybersecurity, product analytics, and BI are converging. A CEO recently asked him why he couldn’t “kill Tableau, Looker, DataDog, Grafana, and QuickSight” in favor of a single system. In my opinion, it doesn’t need to be a single tool, but it should feel like a single interface.

Most common today: a chat prompt that autonomously spawns ingestion, transforms data into marts, and surfaces a dashboard or web app, running end-to-end analytics without the user ever thinking about the layers underneath.

But only the speed of faster building with AI won’t get us there. Amdahl’s Law still applies, as Jeff Dean rightly said in his talk with NVIDIA’s Bill Dally:

An AI agent can run 50x faster, but if the tools it depends on were designed for human speed — slow query APIs, brittle CLIs, unversioned metrics — the overall gain collapses to 2-3x.

That’s why the primitives behind BI get more important as agents get faster, not less. OLAP needs to be faster, metrics need a reliable API, ETL needs to be composable. The bottleneck shifts from the model to the infrastructure it runs on.

And when agents do take over the analyst workflow, spawning parallel queries, discarding dead ends, surfacing the interesting slices, they’ll expose something BI practitioners have known for years: the hard part was never the visualization.

It was always the semantics beneath — the governed metrics, the trusted definitions, and configs that were verified by an actual human being. Agents will just make that gap impossible to ignore.

BI primitives are Infrastructure for AI

Wrapping up, BI was never about dashboards. It was about making sense of a company’s data, connecting the source data into something a human can understand, efficiently reusing existing metrics, and governing definitions.

The dashboard was just the visible surface. What survives the hype cycle, from the unbundling of the modern data stack to the rise of AI agents, are the primitives: metrics, semantics, ownership, trust.

The AI era doesn’t kill that need. Agents hallucinate without a strong foundational semantic layer or verified human constraints. Non-deterministic chat interfaces collapse without business-wide, agreed-upon definitions. The maintenance problem doesn’t disappear either when you generate faster. It compounds the problems and bottlenecks for senior engineers at a company.

BI-as-Code, versioned dashboards, and a governed interface aren’t opposite to the AI future, but a necessary foundation that makes working with it easier, not only for AI systems but also for humans in the loop.

–

If you enjoyed this, there’s further related reads that might be interesting to you:

• Data Modeling for the Agentic Era: Semantics, Speed, and Stewardship
• Has Self-Serve BI Finally Arrived Thanks to AI?
• BI-as-Code and the New Era of GenBI

Full article published at Rilldata.com - written as part of my services

You might use AI agents all day long, parallelizing them with AI orchestrators like Agent Teams, Gastown, tmux, git worktree, and AI-based IDEs, but in the end, you just coordinated an AI. You still have to learn what it created, understand it, check for hallucinations, and verify that it built the right thing. We’ve all become senior reviewers, more exhausted than before, with less of the work that made this fun in the first place. Meanwhile, we are also more distracted than ever. No time to think, with Copilot, Grammarly, or something else constantly asking and suggesting.

This series interviews real practitioners to give you the best tips on how they use AI in their data work today, extracting as many patterns behind them as possible. The article is structured in four parts: (1) how Mark is using AI, (2) what he has learned working with it, (3) what he is specifically using it for, and (4) what he thinks about AI in general and the future.

Introducing the Guest: #1 Mark Freeman

The start of this series is none other than Mark Freeman. He is currently the Head of DevRel, Employee 1 and GTM at Gable. Mark has gone through three career roles as clinical researcher, data scientist, and data engineer, which is helping him greatly in his current position to navigate the unknown of generative AI. We’ll go more into it later.

Mark has also co-authored a book with O’Reilly about Data Contracts (with Chad Sanderson and B.E. Schmidt), and is helping build Gable with the best possible data flows and data quality for enterprises.

[!abstract] TLDR

To set the stage, in this interview we talk about how to use Spec-Driven Development workflow with Claude Code and agent teams to produce high-quality, reproducible outcomes. We cover Mark’s use of ExcaliDraw diagrams and JSON schemas to define requirements upfront, how he parallelizes agents with tmux to compare outputs, why AI benefits senior engineers more than juniors, and where he sees data engineering heading.

How Mark’s Using AI

Let’s start with the general setup Mark uses when working with AI, and how he uses generative AI.

How Mark Changed His AI Workflows

I asked him: “Since you’re building a company in the data contract and quality space and have written a book, how has working with AI changed how you use AI at work?”

Mark has been in the data space since 2018 as a clinical research analyst and a data scientist since 2019. In 2022 he shifted over to data engineering, and in 2023 joined Gable to solve the problem of applying data contracts. He was very early in NLP with the major ML project he worked on back in 2021.

He remembers the early days in 2023 when ChatGPT hallucinated and when he used generative AI for the first time. Very much as a chat window co-coding companion, asking them architecture questions and general questions about the code at hand. Fast forward to 2024 and 2025, generating more code, but not full programs and projects, but by function - trying to narrow down the scope.

And then in late 2025, Claude Code came around, and changed the game with better models that could autonomously solve problems for a longer period. And at the same time, everyone provided more CLIs to empower the CLI-first workflow of Claude. Mark started building by giving it instructions, pointers to docs, schema, etc., and letting it independently build data-related work and go fully agentic.

Mark’s Spec Driven Workflow

Mark has figured out a very well-working approach that helps him create reproducible outcomes. Not focusing on solutions, but on how the tool works as he relentlessly specs and defines what he wants with the Spec Driven Development (SDD) approach, inspired by Esco Obong and how he used it at Airbnb. He uses the GitHub-provided spec-kit, which is a toolkit to help you get started with Spec-Driven Development.

I hadn’t heard of it, and when checking it out, it’s super well documented and integrates 1:1 into Claude Code (and many other AI agents), meaning you can use slash commands within Claude and define specs with the help of an existing git repo including docs and code such as:

• /speckit.plan: Execute the implementation planning workflow using the plan template to generate design artifacts.
• /speckit.tasks: Generate an actionable, dependency-ordered tasks.md for the feature based on available design artifacts.
• /speckit.specify: Create or update the feature specification from a natural language feature description.
• /speckit.analyze: Perform a non-destructive cross-artifact consistency and quality analysis across spec.md, plan.md, and tasks.md after task generation.
• /speckit.clarify: Identify underspecified areas in the current feature spec by asking up to 5 highly targeted clarification questions and encoding answers back into the spec.
• /speckit.checklist: Generate a custom checklist for the current feature based on user requirements.

You can define these on a per-project basis, or have some of them defined as a general spec in your ~/.claude/ folder. The outcomes are Markdown files that hold dedicated specifications, based on your goals that can then be further edited and updated based on your iterations.

Working Product-Focused

This helps Mark to focus on product scenarios and predictable outcomes instead of vibe coding every piece from describing his principles from scratch, he continues.

He goes from ideation through specs to dedicated tasks. He likes to always start with an ExcaliDraw diagram, defining more of the flow diagram, rather than architecture or other overviews. For data schema and interface definitions, he defines data structure next to the relevant flow diagram, as ExcaliDraw is JSON, these can be easily integrated. Schema definitions describe accurately what’s needed based on stakeholder discussions and his needed requirements.

He then passes that diagram back to Claude Code and iterates on the data model and his key assumptions. Mark takes a lot of time in this process. He will spend hours, days or even weeks in this stage, updating and refining these specs, specifically giving clear and exact information about data schema, tools to use, and architectural choices that he knows as a senior engineer that he wants and needs to have.

This is also where years of experience make the difference.

Using TypeScript for Data Schema Enforcement

An interesting discovery Mark made is that he started using a programming language new to him, TypeScript. Similar to Wes McKinney’s From Human Ergonomics to Agent Ergonomics, where he states that “Python Was Built for Humans, Not Agents” and argues that he is using GoLang and Rust for agent work, as it’s a better language for agents with minimal dependencies and shorter/better types.

Mark ended up using lots of TypeScript, not because he was familiar with the language, but because it’s mostly what his work and that of a data engineer requires: defining data types. Enforcing them, quickly verifying across the data pipeline that we don’t get an error before pipeline runtime. Saving a lot of time and upping the quality.

What Mark Has Learned Working with AI

Over the years, Mark has changed his workflow. In this part, he shows how he uses agentic agents with tmux and how he reviews and checks the outcome.

Agent Parallelization and Executing Them: Teams and Tmux

After all the specs and focusing on them once, he uses agents to implement the specs and Claude uses the feature called Agent Teams (which can be activated in Claude settings.json with CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS).

The cool thing about agent teams is that they let you coordinate multiple Claude Code instances working together. One session acts as the team lead, coordinating work, assigning tasks, and synthesizing results. Teammates work independently, each in its own context window, and communicate directly with each other.

Mark spawns multiple agents using iTerm2 and tmux, which I heavily recommend for agent work (also check Zellij for an easier start), and the agent teams feature will automatically open the additional terminals in separate panes:

It shows Claude self-orchestrating his own team. Think of it as similar to Gastown, Agor, and other AI orchestrators, but integrated into Claude.

Mark’s workflow with agent teams is deliberately outcome-focused rather than code-focused. Once the agents complete their run, he checks the result against the original specs and JSON schemas, not the code itself. The only thing that matters is whether the outcome does what was defined.

Is Reviewing Code Still Needed?

The tough question was whether Mark still reviews code, especially when Claude can generate more of it in a minute than we can ever review. Mark said: “Not locally or on unimportant projects where I’m exploring the limits and potential traps of these powerful tools.”

But for production pipelines or when customers asked him specifically for his opinion, he said:

Along with the wider industry, we are figuring out how to use AI safely at scale.

Also at work when they have mission-critical services such as in a bank, you can’t just vibe code something. It comes down to use-cases, he said.

Besides use cases, he tried different ways of reviewing. First he tried a sophisticated process where the above agents would create PRs and he would then comment on these with improvements and changes. The agents would then read them and integrate the given feedback and continue the process. But even that workflow made him too much of a bottleneck. It wasn’t scalable enough.

Mark searched for other ways to work with it.

Outcome-Driven Reviews: And Starting from Scratch Again

What he does now is assess outcomes instead. After all the rigorous time in speccing, he tests the result by running the pipeline, creating tests, or checking the code manually the old-fashioned way.

The key mindset shift here is that the first build is deliberately treated as throwaway. It’s requirements exploring via building. You implement the spec once, learn what you got wrong, and expect to discard it.

That’s why he tests the outcome. And once tested, he might have gotten new learnings that he could have only gotten through implementing or with actual tests. That’s when he will feed these learnings back to the specs and update initial requirements, and start all over again, from scratch, letting the agent create a new outcome based on the updated specs. The cycle is: spec → build → assess → improve spec/assumptions → repeat.

This way, he has an approach with a very deep and exact iteration, almost deterministic, where he can re-run the agents with updated feedback and requirements, and get the same or similar outcome with the added updates, because of the spec-driven approach and the structured approach that spec-kit delivers, and the dedicated way he defines his requirements, which won’t just be hallucinated as different inputs, end-to-end.

Though this can always happen, this approach served him very well, with a high-quality output he can trust, and a qualitative way to approach a complex problem with the help of agents.

If the outcome meets the quality he expected and it does what he wants, he goes to internal stakeholders to get feedback from them. And then the same process again, updating specs, fixing requirements errors or possible wrong assumptions, and off the agents go again.

Tests and Quality Gates

Tests and QA he writes manually. This is another way to make sure the outcome meets his expectations. Most important is the value, he says:

Value first, then outcome and then worry about other things

If it’s not turning out to be valuable to the stakeholders, he wants to avoid spending more time. That’s why the agent iterations and building something “quickly”, with rigorous specs and definitions in place, worked well for him so far.

Senior vs. Junior: Working with AI

We move on to an interesting discussion of whether AI helps senior engineers or juniors more. Mark says (he also wrote about it) that AI helps more senior engineers, as seniors “understand the trade-offs of tech debt”.

He says further that in AI iterations, we move much faster, generating legacy code and architecture constructs in days and weeks, instead of years. If Mark iterates with the spec-driven design explained above, there are multiple different architectures generated, some of which might have been bad from the very beginning.

As a senior, he thinks that we can give the right guidance from the very beginning and exclude bad outcomes and early “legacy code”. No doubt, there will be code and architecture to be adapted, too, but if you lack experience, you basically have no chance of knowing.

Framework and Architectures Are for the Experienced

Mark mentions that at Gable, he is building something from scratch. Let’s say we are at iteration v4: deep technical architectures are coming up, to choose an Apache Kafka infrastructure, define your schema in JSON or Avro, or use Parquet.

These decisions can only be made with experience. Sure, agents will give you a good middle ground, and with research they will potentially choose the right solution for the current problem. But how do you know what’s the best solution for your given business problem? If you have built multiple data platforms and have seen many companies, you just know some of these things or developed an intuition for what’s needed.

In combination with the agents, it’s just a much better tool for seniors than for juniors who need to more or less blindly trust the assessments the agents made. The quality of outcome depends on frameworks and architectural choices, accumulating legacy code early if a big architectural component is chosen wrong.

In a related but further way, the knowledge is like a linter in an editor that knows things ahead of runtime. It can detect wrong choices directly.

What Mark is Using AI for

Besides the already discussed use cases of general workflow and reviewing outcomes, I asked him about how he uses AI at work, working with data contracts and the non-deterministic outcome of AI, for example.

Integrating AI into Data Contracts

As an author of a book on data contracts, and working in the business, one of Mark’s priorities is to somewhat safely use AI agents to either verify contracts or help define them, if in any way possible.

As data contracts are written definitions between two parties, mostly written in YAML or JSON, it’s a good medium to iterate on, where agents, humans, and all stakeholders can work on specs that can be versioned. Mark says his focus is on evals, specifically for assessing how well an agent completes a specific task, built around Gable’s products or internal workflows.

The main goal of evals is to more confidently know that what AI shipped is any good. Similar to stewardship in Master Data Management (MDM), where humans in the process verified if the data quality was met, with AI generation we need a similar process at a faster pace.

That’s also where he draws on his clinical background with an outcome-driven approach, comparing 200 observations from end-to-end coding agent simulations and assessing results against defined criteria. At Gable, they create a Code Graph that helps them get a skeleton view of the full data flow in code, without running any code. Connections, context, and business operations are expressed as code to be verified.

His hypothesis is that with agents at scale, we can gather datasets of behaviors such as logs of data pipelines, network logs, and other information such as agent trajectories and check based on them whether the data pipeline is compliant, like A/B testing AI Agents with a Bayesian Model. This has yet to be proven, but the hypothesis is strong.

Deterministic and Non-deterministic Work in Data Engineering

When asked about his thoughts on functional data engineering where usually jobs are reproducible and restartable with new logic/source data, and how he sees the determinism with AI work (which has a different outcome every time), he said something interesting.

He said non-determinism is a benefit. That’s why the setup is specs written in markdown, combined with configs and JSON that are specific, providing precision and accuracy. If anything goes wrong or not according to plan in the generation phase, he can just change the specs and achieve this determinism by spec-driven development.

But there are still some disadvantages from running non-deterministically, that’s why he still does tests and comparisons manually, and checks visually whether everything works when running the pipeline.

What Mark Thinks about AI

When talking about the future, learning with AI or where it leads, or also when not to use AI, is what we discuss here.

When not to Use AI

Starting with when he is not using AI, and when it’s potentially cheaper or better to do it manually, his answer was:

Requirements finding in an important project, again depends on use cases. For small non-personal projects, not a problem. But requirements need to be defined by stakeholders and come from a real problem

Also, Mark mentioned key decisions for infrastructure code that needs to be stable and reliable. Or if used, he will spend much more time validating that LLM suggestions are correct.

For content online, he noticed that the writing always comes off differently than he would have phrased it. He might give it his insights to check or get feedback, but not the actual writing part.

How Do You See Learning with AI?

There’s also the danger of not learning new things, and getting overwhelmed with constant stimulation, potentially getting slightly addicted. I asked Mark if he sees a problem in using agents and LLMs that would prevent us from learning new things as we are just cruising on auto-pilot.

Yes, he agreed. He calls it: “Claude code slot machine”, or “Lab rat”. “Getting your dopamine hit beyond usefulness” is how he would phrase it. He also thinks that this addictive behavior doesn’t exist randomly. He thinks it is intended for us, the users, to use and spend more tokens (ergo money for them).

[!note] Pseudo Work

Shipping lots of code with AI can feel like deep work, but if you’re not learning in the process, it’s pseudo work. Problem-solving skills in adults are already declining, and even studies showing short-term learning gains with AI find that beyond 8 weeks, the effect reverses as over-reliance sets in.

The Future of Cloud vs. Local Model

My closing question was where things are heading, and whether self-healing data pipelines would be a thing. When some say that “Unironically, Rick Rubin is the future of work” (where AI replaced a team of analysts, a strategist, a designer, a project manager, and a few weeks of work in minutes), the same goes for data analytics and data engineering.

Mark explains that when he was a data scientist, getting a nice histogram in Matplotlib or Seaborn took hours. Today he gets that for free, as an afterthought. He has built applications that pull leads from Hubspot, extend and aggregate data through RAG using APIs and pipeline logs, and for a board meeting just generate a static HTML page (with an export to CSV 😉). A custom-made visualization at your fingertips. That’s the future, he says. Because below the visualization, there’s a semantic model as the base. No one wants to open one more app, so based on well-defined semantics, AI can one-shot the visualization and integrate into existing workflows.

On the local model side, another future he sees (and is exploring himself) involves models running on a dedicated machine while he’s away. He said the future is not about how powerful the models are, but how many iterations your spec has gone through. You run them until they are correct. You can also use RAG techniques to augment the model with your own notes and skills, one local model custom-made for you:

You can’t compete on compute, but you can use the factor of time, iterating multiple versions for a specified problem, and choosing the best one. Exactly what clinical research is doing and what he learned in his early career comparing studies.

An interesting bleeding-edge area is running agents optimized for concurrency, chunking tasks and parallelizing them with smaller compute resources instead of one big model. Abi Aryan is doing GPU research in exactly that field, and Mark recommends starting with this post. While companies are paying 10x or more for cloud compute, local models with lots of iterations are increasingly feasible, and the economics are starting to make a strong case for them.

Next Interview

I hope you enjoyed this interview with Mark. Huge thanks to Mark for taking the time to speak with me and for sharing his experience with all of us. Follow him on LinkedIn and his Course on data quality and check out his book, its repo, and much more.

There are three more interviews already lined up with great guests, so please share feedback, questions you might want to ask or just your experience on how to work with AI in the data space. We’re all in this together, figuring it all out. The more we can learn from each other, what’s important, and maybe also what’s not, the better.

So stay tuned for the next interview.

Full article published at MotherDuck.com - written as part of my services

When Rill bet on YAML dashboards and CLI-first workflows in 2022, they weren’t thinking about AI agents. Neither was MotherDuck when they built serverless DuckDB around the thesis that most data fits on a laptop. But it turns out, what is developer-friendly is also agent-friendly, with the needs of readable code, fast engines, and deterministic semantics.

Times are shifting rapidly toward CLI-first development. You know that’s true when even email and calendar get their own Google CLIs. So why not have CLIs for your business metrics too?

This is what Rill and MotherDuck provide, including excellent developer workflows with a local-and-CLI-first approach, focusing on a developer-friendly interface and empowering users. Both work great on local laptops but can easily scale to the cloud, backed by a serverless data warehouse..

The convergence of embedded analytics engines (DuckDB/MotherDuck), declarative BI-as-code (Rill), and AI agent protocols (MCP) is creating a new architecture for business intelligence, one where dashboards become code, code becomes agent-readable, and analysts shift from clicking to prompting. And with 75% of cloud data warehouse queries scanning less than 1 GB¹, this opportunity is great for agentic BI. In this article, we look at how we build agentic-friendly and local-first analytic stacks with MotherDuck, Rill, and agents.

[!note] End to end examples later in the article

Later in the article we go through three different examples of how this can work, including GitHub repos and code examples, if that is something that’s of interest to you.

Why These Two Tools?

Let’s start with why do we use MotherDuck and Rill for agentic-first data tasks? As Ghanshyam Chodavadiya from SWYM says:

[!quote] Quote on why SWYM use Rill with MotherDuck for their AI-native media decision platform:

[..] Rill lets us encode business context directly into our BI layer. Combined with MotherDuck and the Rill MCP client, it gives us flexible data control while powering automatically generated client dashboards and AI-driven insights.

Both MotherDuck and Rill use a sophisticated architecture that focuses on developer workflows and scales from local development with declarative configuration to cloud (e.g. with rill deploy, or md: instead of md) or even embeds into your data CI/CD or agents pipeline, very easily. All of these reasons make them suitable for modern data requirements, where we need to iterate quickly but still have a strong foundation.

Local-first Approach with DuckDB

Both tools start from a local-first approach with DuckDB as the foundation.

For example, the Local-First principle, that was tackled by Ink & Switch and its community compares different strengths of local workflows meant for files, but also applies to data workloads. Even more so with AI agents, which can read the context from these projects and enhance easily with the use of strong CLIs that are available on the command line.

Or reading the Small Data Manifesto by MotherDuck that says “Think small, develop locally, ship joyfully”. If you enjoy some of these principles, this data stack with DuckDB/MotherDuck as the warehouse or real-time analytics storage and Rill as an interactive, fast, and beautiful BI tool will suit you well.

MotherDuck works seamlessly through the DuckDB CLI, whether it is to connect through their serverless database in the cloud (connect with duckdb ':md') or to open a fully fledged notebook environment locally with duckdb --ui (try it).

With Rill’s YAML-based dashboard and metrics layer, and a powerful CLI, you can transform any of your data into a blazingly fast dashboard locally (run rill start), or from anywhere with the data on MotherDuck. Let’s explore both in more detail, and show how users use it, and provide an example for you to get your hands on.

What Is MotherDuck?

Before we go into the hands-on examples, let’s answer the question of what MotherDuck and Rill are. And what’s the difference from DuckDB and what do they bring to the table?

In its essence, it’s a DuckDB-powered cloud data warehouse that scales to terabytes with ease. Just as Turso hosts SQLite, MotherDuck hosts DuckDB in the cloud, serverless for you. MotherDuck has great relation to DuckDB Labs, the company behind DuckDB and the DuckDB Foundation.

MotherDuck integrates well with DuckDB, but you can also just run DuckDB locally without it and manage your server yourself, open some ports, make it scale automatically if more queries come. But you’d need to create an orchestration that scales out, handles OOM, servers, etc. So instead, MotherDuck provides all of it with simply pointing your local database to connect via ATTACH 'md:' compared to directly reading from a local DuckDB database (duckdb path/to/file/db.duckdb) or parquet files (FROM nyc.parquet or FROM read_parquet('test.not-ending-with-parquet')) that only you have access to.

The simplest way is to initially upload data to MotherDuck once, and then have access to the data from anywhere (see example later in the article).

[!example] How to Upload data to MotherDuck from Local Storage

You can basically use different ways of synchronizing local DuckDB to MotherDuck via COPY FROM DATABASE, CREATE OR REPLACE DATABASE ... FROM '<path>', from Parquet files, to using Python files - you can find more details at sync-duckdb-to-motherduck.

Besides simply replacing local DuckDB with a data warehouse like MotherDuck, MotherDuck has implemented a specific architecture called dual execution. It’s built on top of their 1.5-Tier Architecture. It’s a novel 1.5-tier architecture powered by WebAssembly (Wasm). Unlike the more traditional 3-Tier architecture that operates between the client and server, the 1.5-tier directly returns the request in the client (browser), reducing latency for server requests and network round trips.

Traditional applications are built on a 3-Tier Architecture, which requires several intermediary operations to run between the end user interface, server, and underlying database. MotherDuck’s 1.5-tier architecture has the same DuckDB engine running inside the user’s web browser and in the cloud.

The developers can move data closer to the user to create analytics experiences that run instantly with the benefit of still scaling with MotherDuck as the backend. Check their CIDR paper on DuckDB in the cloud and in the client, on how this works in detail.

What Does the Dual Execution Do?

Since the initial paper, the dual execution has evolved and makes MotherDuck more than just “DuckDB in the cloud”. When you ATTACH 'md:' locally or in the web, you get a two-node distributed system that automatically routes query stages to wherever they run best.

An example with using dbt: In your sources.yml of dbt you can simply define dev with DuckDB and prod with MotherDuck like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

your-project:
	target: prod
	outputs:
		dev:
			type: duckdb
			schema: project_dev
			path: "path/locally.duckdb"
			thread: 1
		...
		prod:
			type: duckdb
			schema: project
			path: "md:prod_project"
			thread: 1
		...

Smaller data gets processed locally with millisecond response times and if needed, can extend to run in the cloud, using cross-environment joins that transfer only the necessary intermediate results. As a user, you won’t notice the difference in using simple DuckDB.

This uses your laptop’s power with DuckDB as a first-class compute node. As MotherDuck CEO Jordan Tigani put it: “Laptops these days are extremely powerful and you can get answers in a handful of milliseconds, whereas if you had to ask a cloud service, the initial request wouldn’t have even gotten there.”

In a way, MotherDuck is a lightweight alternative to Spark for single-node or moderately-sized analytical workloads (it does not support distributed, multi-node processing on massive datasets), but it’s far easier to set up, has no cluster management, and scales to terabytes. Without the setup cost or the operational burden for tasks that don’t need the massive scale Spark provides, you get an out-of-the-box data warehouse that handles scale very conveniently for us users.

[!example] One more advantage is multi-user collaboration

DuckDB is single-writer, and MotherDuck is what unlocks the “multiplayer” angle with its integrated notebooks (which are also available locally with duckdb --ui, but not shareable on the web).

Why Rill?

So what does Rill bring to the table, and why do they work so well with MotherDuck?

If MotherDuck gives you a data warehouse that feels like a local database, Rill gives you a BI tool that feels like a code editor getting you up and running with a single binary you can start with. The name Rill is from an old English word and meaning “stream”, and it has strong templates integrated that scaffold any BI requirements in seconds. Both MotherDuck and Rill are built on the conviction that focusing on developer experience will help data teams implement great data solutions that not only work, but are fun.

Rill’s core idea is simple: define your entire BI stack (data sources, SQL models, metrics (total_revenue: sum(amount)), dashboards) as YAML and SQL files in a Git repository. Start simply with one command (curl https://rill.sh | sh), run rill start, and you have a local development environment backed by an embedded DuckDB instance delivering sub-second queries. Push to Git, deploy to Rill Cloud, and your dashboards are live. The same declarative files, the same SQL, the same metrics, just a different runtime.

Moreover, this “BI-as-Code” approach turns out to be exactly what makes Rill a natural companion for agentic workflows, because all artifacts for data and BI are defined declaratively and locally, any agent can use them for context and build autonomously on top, while still letting the user verify that everything is correct and works by quickly running the Rill CLI locally or in a CI/CD pipeline.

Rill embeds DuckDB under the hood. Connecting it to MotherDuck requires nothing more than a YAML connector config pointing at md:my_database with a token property such as `token: “{{ .env.CONNECTOR_MOTHERDUCK_TOKEN }}”.

The SQL models are identical, meaning no syntax changes, no migration, no new query dialect to learn. The only thing that changes is where the data lives, but that has no effect on the user experience.

Rill was built around the idea of:

That instead of hiding business logic in a proprietary GUI that only humans can click through, you make it readable code that anyone, or anything, can openly read and reason about.

Rill with its YAML dashboards and a CLI-first workflow has positioned itself perfectly for working with AI agents. That wasn’t planned or foreseen, but it turns out that context is king and that tools designed for developer simplicity are exactly what agents need with readable definitions and fast deterministic engines.

The result is a stack where your metrics are a source of truth you can version, audit, and feed directly into an agent’s context window, and where switching from a local DuckDB file to a serverless cloud warehouse is a one-line change.

Metrics and SQL

We all probably agree that metrics are the key to codify business knowledge, giving us all the benefits of a software design approach (versioned, automatable, testable, etc.) and automating metrics with an agent, while still being able to define complex metrics ourselves if needed. And an agentic-friendly environment where agents get their concise context and collaborate with a domain expert and human in the loop.

It’s good to know that Rill provides an integrated Metrics Layer. It’s excellent as it’s just a YAML file too. Meaning you can integrate it into other notebooks, or data apps easily, but also just with Rill for building multiple dashboards, conversational analytics, and canvas on top of a unified metrics layer.

Instead of integrating complex metrics multiple times in different dashboards, we can just reference the metrics layer. Besides the metrics, we need fast, instant responses, even more so when we let agents work autonomously. We can’t wait 5-10 minutes for a simple question until all research through the agents is done.

[!info] Metrics SQL: A language built on top of SQL

Additionally, Rill offers a dedicated query language that extends on the strength of SQL, called Metrics SQL. It’s a dedicated SQL dialect designed for querying data from Metrics Views.

I wrote more at a SQL-Based Semantic Layer, but it helps to simplify your SQL as it empowers the SQL language, or learn more about the philosophy from Mike, the CEO of Rill, in an insightful podcast with Joe Reis where they talk about the future of dashboards and how agents and navigating the new era of BI and analytics works.

Conversational BI: Rill Turns Dashboards into Code (and Code into Agent-readable context)

When looking at Conversational BI and its benefits, we can say that “Conversations can generate code, and code generates insights”. Code is the best abstraction, and with agents, we can easily make it available to non-programmers.

Code as the abstraction layer in most cases. But why? Because if you create a hard-coded interface language, or an API, you can only do what you need. With code (usually Python, or SQL in this case too), we can do much more. We can use all the functions of the language versus the implemented API. It’s easier to maintain, and also automate.

RudderStack reinforced this narrative from the infrastructure side:

Most of today’s commercial data tools are designed for humans, not for automation. Their primary interfaces are web dashboards, which are convenient for analysts, but opaque to code.

This means if we want agent tools to analyze our code base, we need to let them access our code, or in the case of Rill, declaratively defined dashboards, metrics in the metrics layer, and data sources.

We can do even more when we use the chat interface to interact, making it usable for humans again, making it usable for humans again by using natural language as the primary interface. That’s where Rill offers an extensive integration with agent workflows through generating a dashboard based on existing sources, models, and defined metrics in the metrics layer:

These features are also integrated as Conversational BI, letting you explore your business numbers with the interface of a natural language and chat. With Cursor and agentic code-like suggestions, but referring to pre-defined metrics for asking specific questions:

Here’s a full video of what you can do with it.

The chat interface provides charts that can be further explored or integrated into your dashboards. What I like most is that in the responses, you can just click on them (e.g. 1.) and it will appear as a pivot table in (2.) where you can dig into more details by adding more dimensions and metrics:

Showcase of how interactive dashboards are created on the fly, to explore and open in a pivot table directly inside Rill. The links are clickable, see a short video of how that looks in action:

[!note] You can also use Claude integration with MCP

Rill’s MCP can also be used, see Showcase of Rill MCP with Claude Desktop - YouTube. Or Cursor if that is your preferred AI-based IDE at BI-as-Code and the New Era of GenBI Demo - YouTube.

Let’s now look at different analytics use cases and implementations with these handy features combined with MotherDuck as the backend.

MotherDuck + Rill in Action: Three Examples

Let’s now look at actual implementations with these features combined. We’ll start with how to connect Rill to MotherDuck, then walk through two open-source examples you can try yourself, and finish with a real-world customer showcase.

Connecting Rill to MotherDuck

Since Rill already embeds DuckDB, connecting to MotherDuck requires only a four-line YAML connector config with a token and a md: path. Add a motherduck.yaml to your connectors/ folder:

1
2
3
4
5

type: connector
driver: duckdb
mode: readwrite
token: "{{ .env.CONNECTOR_MOTHERDUCK_TOKEN }}"
path: "md:my_database"

Compare that to a local DuckDB connector:

1
2
3

type: connector
driver: duckdb
dsn: "my_database.duckdb"

The only difference is token + path: "md:..." instead of dsn. Set the token as an environment variable (see Rill docs for details), and your SQL models, metrics, and dashboards work identically — whether the data lives on your laptop or in MotherDuck’s cloud.

In 2025, Rill significantly strengthened its native connectivity to enable zero-copy, blazingly fast analytics without moving data, making this connection even more seamless.

Stack Overflow Developer Survey: Zero-Pipeline Analytics

The simplest way to experience MotherDuck + Rill is with data you already have. Every free MotherDuck account ships with sample_data.stackoverflow_survey.survey_results² — 600k+ professional developer responses from 2019-2024. No ETL needed.

The motherduck-rill project builds a complete analytics stack on this data: 4 SQL models (staging, technology usage, developer profiles, database analysis), 3 metrics views with 17+ measures, and 3 canvas dashboards — all as pure SQL + YAML in a Git repo. No Python, no orchestrator, no data pipeline.

1
2
3
4
5

git clone https://github.com/sspaeti/motherduck-rill.git && cd motherduck-rill
cp .env.example .env
# Add your MotherDuck token to .env
rill start
# Open http://localhost:9009

That’s it. One command and you’re exploring which databases are most desired in the US, which languages pay the highest salaries, or how AI adoption shifted across years — all backed by MotherDuck’s serverless cloud.

Answering the question of “Which databases are most desired in the US according to Stack Overflow”:

The SQL models use standard DuckDB syntax throughout. For example, the database_analysis model unnests semicolon-separated survey responses into one row per database per relationship type (used, admired, desired), then the metrics view aggregates them with COUNT(DISTINCT ResponseId). The same SQL runs locally via embedded DuckDB or via MotherDuck — the connector config is the only difference. This is what BI-as-Code looks like in practice.

The above example could be created based on a couple of simple prompts, as Rill’s definitions are all local and the data accessible through MotherDuck through DuckDB CLI.

Multi-Cloud Cost Analyzer: Production-Grade Connector Switching

For a more production-like setup, I use the cloud-cost-analyzer I’ve built in a previous edition to visualize your costs from different hyperscalers with ClickHouse, and now added MotherDuck.

This project shows how MotherDuck fits into a real data pipeline alongside local DuckDB and ClickHouse. Same dlt pipelines, same Rill dashboards where I just added a different destination to dlt. Zero pipeline code changes were needed.

The key insight: MotherDuck uses DuckDB SQL syntax, so the SQL models share all functions with local DuckDB. Only the FROM clause differs — read_parquet('...') locally vs schema.table on MotherDuck. The Rill SQL models use a 3-way conditional to switch to work around these small differences between connectors:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

{{ if eq .env.RILL_CONNECTOR "motherduck" }}
SELECT CAST(SPLIT_PART(identity_time_interval, 'T', 1) AS DATE) AS date, *
FROM aws_costs.cur_export_test_00001

{{ else if eq .env.RILL_CONNECTOR "clickhouse" }}
SELECT toDate(splitByChar('T', identity_time_interval)[1]) AS date, *
FROM aws_costs___cur_export_test_00001

{{ else }}
SELECT CAST(SPLIT_PART(identity_time_interval, 'T', 1) AS DATE) AS date, *
FROM read_parquet('data/aws_costs/cur_export_test_00001/*.parquet')

{{ end }}

Notice how MotherDuck and local DuckDB share the exact same SQL functions (CAST, SPLIT_PART) — only the FROM source changes. ClickHouse needs its own dialect (toDate, splitByChar). This is the practical advantage of MotherDuck being DuckDB in the cloud: your SQL models stay the same.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

git clone git@github.com:ssp-data/cloud-cost-analyzer.git
cd cloud-cost-analyzer
uv sync

# 1. Add your MotherDuck token to .dlt/secrets.toml
#    [destination.motherduck.credentials]
#    password = "eyJ..."

# 2. Add token to viz_rill/.env
#    CONNECTOR_MOTHERDUCK_TOKEN="eyJ..."

# 3. Load data into MotherDuck (same pipelines, different destination)
make run-etl-motherduck

# 4. Start Rill dashboards backed by MotherDuck
make serve-motherduck

Under the hood, DLT_DESTINATION=motherduck tells dlt to write to MotherDuck instead of local parquet files. The same data is then also visible in the MotherDuck web UI for ad-hoc querying alongside the Rill dashboards.

This pattern — make serve-motherduck vs make serve vs make serve-clickhouse — shows what switching from local to cloud looks like in a CLI-first stack.

Driotech: Agentic Analytics in Production

Beyond my own demos, Salomon from Driotech showcased how they use Rill for agentic analytics with client data using MotherDuck in combination with Airbyte, dlt, BigQuery and dbt.

In his webinar on empowering businesses with agentic analytics, he walked through a B2B sales use case that highlights exactly the principles we discussed above.

His key takeaways align perfectly with the MotherDuck + Rill thesis:

1. Metrics as the foundation: Before any AI agent can work reliably, you need clearly defined KPIs and a single source of truth. “If you feed any AI agent with a mess, you’re going to end up with an even bigger mess.” → This is exactly what Rill’s metrics layer provides: versioned, agreed-upon definitions in YAML.
2. Governance as guardrails: The agent doesn’t hallucinate on business concepts because the semantic model constrains what it can query. When asking “what are our shipping costs?” → The agent looks up the exact metric definition, avoiding guessing from raw data.
3. From reactive to proactive: Beyond just answering questions, the demo showed creating alerts (“if customer orders drop to zero in 14 days, email the account manager”) and scheduled reports → Pushing insights to stakeholders automatically.
4. Code-defined dashboards: Salomon explicitly called out that Rill’s code-based approach means “AI agents are able to build the dashboards for us” → Because the language of AI is code, and Rill’s dashboards are just that.

The above webinar reinforces a central point we’ve discussed above that conversational BI can be used today in Rill. When you have a solid semantic foundation (metrics in Rill), a scalable backend (MotherDuck), and a code-first workflow, the agentic layer becomes practical, trustworthy, and something you can deploy to real users today.

What’s Next, Dashboards no More?

Seeing these examples, and where we are heading with agentic BI, one might ask now, do we even still need dashboards? We can just ask the chatbot?

I agree with Mike’s argument that: “No, well-crafted, and especially operational dashboards, they will never go away”. Because a visualization can provide so much more condensed information in a couple of seconds that a chat never will, or would require many back-and-forth chats.

There’s also a difference between “a” dashboard and the dashboards. E.g. more than half of dashboards are just exploratory in nature to quickly explore the business or data. Additionally, the sales dashboards that someone spent weeks or months, sometimes years for large companies, to perfect and ensure the correct numbers, deriving key decisions for the business from it or defining a sales rep’s salary as how much they sold usually is tracked in a dashboard as well.

One more thing that chatbots can’t replace is so-called “drilling down”, and using data as a REPL with pivot tables - as they never die. Quickly drilling down to the lowest details and back up to the aggregated data within seconds, simply dragging and dropping dimensions and measures around.

[!tip] People don’t know what they are looking for
Benn Stancil argued, “the challenge with data exploration is not that people don’t have the ability to manipulate data; it’s that they don’t know what they’re looking for.”

Natural Language Interface: Convenient, but Inaccurate?

With agents, we can use natural language as an interface to input domain knowledge, and agents will do the technical translation and implement it in such a code and declarative-first approach, where the context is clearly and distinctly defined - much more than natural language, which contains lots of nuances and ambiguity.

So with that approach, the agent will put it into deterministic YAML, that can then be reviewed, tested and automated against. So we move from Human to agents to context to iteration and finally, visualization:

Similar to what we discussed in GenBI, iterating much faster than with the traditional, non-generative way:

Self-Serve with BI-as-Code providing the Context

So what’s next? Are we arriving at self-service BI finally? (the never-ending promise🙂).

Agents with natural language solve a big problem that self-serve always strived for: Giving each domain user who is less technical an edge to do self-serve themselves, and potentially even go further and fix the data by prompting the data pipeline to fix the correct timestamp, or update a dbt model with the right table source.

With domain experts doing more developer work with agents, we can combine domain knowledge with coding abilities of agents, bridged by natural language with BI-as-Code providing the semantic context that includes models, metrics and even dashboards in plain YAML.

Context is king for the near future. Everything that can be locally defined, such as Rill’s metrics layer and dashboards, will be so much faster and better built with agents.

But BI is not the only domain noticing this power, Kurt Buhler of Tabular Editor wrote: “CLI tools provide an alternative way to interact with software in a terminal by writing and executing commands. This command-line interface is very suitable for agents.”

SQL, YAML: And why Language Choices Matter

Also, with Rill and MotherDuck we choose SQL as our primary language and interface and YAML as structured format to store. And the language choice matters because the training data for widely adopted languages like SQL is larger, thus LLMs are better at generating BI-as-code in SQL than DAX, LookML, or some obscure language.

Wes McKinney even argues that AI agents are enabling him to build software in languages like Go and Swift, despite his lack of prior experience. He says that “human ergonomics in programming languages matters much less now,” as agents prioritize fast compile-test cycles and frictionless distribution, favoring languages like Go and Rust over Python for new systems. Interested in more, check the conversation with Wes and Mike.

Limitations

The one limitation for the future of data might be the imprecise way of natural language and how we communicate. For example: “give me the analytics for this week?” Did you mean “from today until last week”? Or “full weeks Monday to Sunday”? Or “starting from midnight, or during the day”? So many unknowns and misinterpretations possible.

The other is that the future of data needs to be deterministic and reproducible, to backfill faulty data, but AI agents are the opposite. And that can be challenging.

Text-Based, Local-First: The Architecture Agents Need

The connectors between source and destination are getting more flexible, more fluid, self-healing. Knowledge workers might soon be able to not only figure out the problem, but also act on it directly: “send an email to…” to fix the problem.

As the examples in this article show, we’ve come a long way with Self-Serve BI, and we might already be there. Keep in mind the three pillars: semantics, stewardship, speed as you work in the Agentic Era.

The MotherDuck + Rill story is ultimately about the data industry discovering that the tools best suited for AI agents are the same tools that respect simplicity, transparency, and developer ergonomics.

The “small data” thesis didn’t anticipate the AI agent revolution, but it created the conditions for it: when your data fits on a laptop and your dashboards are YAML files, an AI agent can read, reason about, and act on your entire analytics stack.

The irony is that going back to local-first, text-based, SQL-defined analytics turns out to be the most forward-looking architecture. And dashboards become agents when they’re written as code.

–

If any of these interest you more, also check out related articles around conversational BI, BI as code, and how AI can self-serve us in the world of BI:

• BI-as-Code and the New Era of GenBI
• Data Modeling for the Agentic Era: Semantics, Speed, and Stewardship
• Has Self-Serve BI Finally Arrived Thanks to AI?

Full article published at Rilldata.com - written as part of my services

My prediction, and hope, is that the [[Future of Blogging]] is more connected. Not only one dimensional, like a single sheet of paper, but think of a maze, where you can go in, explore new things to learn.

This is how I built up my Second Brain, and you can see the interactive graph at the end of this blog, connecting all notes and blogs that are related.

This article is based on a recent interview about “Write that Blog”. This triggered me to finally write this piece after collecting 100s of notes related to writing online and blogging in my second brain.

Why I Started Blogging: Learning OSS Tools

A quick note on how I got started. Mainly it was out of curiosity. As a business intelligence specialist with a Microsoft licence, I was more curious about open-source tools that had similar abilities as [[SSAS]], [[SSRS]] that were used at work, even more so, the programmatic first approach to automate things, instead of clicking myself through the UI of older GUI first approaches.

I had some when I lived in Copenhagen, Denmark, which I used to explore and document what I learned. As I already had a domain (sspaeti.com) and experience in web development with weekly party pictures that I ran for many years, but wasn’t active anymore as Facebook and other portals got created, I decided to pivot it to a personal blog - which was very popular back then.

So I started with my WordPress blog and uploaded some scripts and learnings around Microsoft and related automation I learned, and was re-using often. Then I did a deep dive and a series on data warehouse automation tools, which got very good feedback after the initial blogs didn’t go anywhere.

I found myself enjoying the process of distilling knowledge in a compact format, so others, and mainly myself, could learn new topics. The [[Feedback Loop]] was another amazing feeling that I didn’t know beforehand, along the principle [[The more you share the more you get]] - as people were giving me suggestions, new ideas, sometimes criticism. But all to find even more open source tools and interesting approaches.

What Started as a Hobby Turned into a Full-time Job and Business

Writing became one of my favorite hobbies, and I got lots of fulfillment, not the short term dopamine hit, but the long-term [[Deep Happiness]] of learning, getting appreciated by readers, and the process of turning my long taken notes into something more usable for people to share. [[Learn in Public|Learning in Public]] as some called it later on.

I reserved Friday nights in my favorite library in Copenhagen, bought my favorite coffee at Espresso House, mostly a nice cookie or something sweet, and then off for 2-4 hours. Sometimes nothing really good came out, it was hard. Other times I was just trying new tools like [[Dagster]], [[Delta Lake]] etc., and others I was in [[Deep Work|deep flow]] of writing, almost like trance.

The breakthrough came much later, three years in, when I wrote about a new upcoming topic, and how the transition from data warehouse I see, called Data Engineering, the future of Data Warehousing?. This was the first viral post, and popular figures like Dan Linstedt commented on it. It was surreal back then, why would these people read my article?

But it gave me the motivation to continue. Sure I love writing and sharing in public, but not sure if I wouldn’t have people reading it, if I would have continued until today.

[!note] A short journey of how my domain and website evolved

I started my first blog in 2015 - but I was online and registered a domain in 2004. I bought the domain sspaeti.com where my first endeavor was web development with HTML, CSS and PHP. The classic Apache years (fun fact, I still deploy to apache server to this day, but it’s only static HTMLs today :)

From 2005-2014 I ran a local forum and party guide (this was before FB :) and then in 2015 my first data-related post. 2016-2018: Regular blogging on Business Intelligence and data topics. 2019 I started to focus more on open-source data engineering.
2021 I moved from WordPress to GoHugo and 2022 I added the second brain to my website which meant all my notes and blogs were powered by Markdown which led me to share much more as it took me no conversion or work to publish anymore. What I wrote, I could just publish as is on my second brain. To this day, I have ~9000 private notes and ~1000 public notes. And 81 blog posts and some chapters of an early book I’m writing in Markdown too :).

2023 I changed the domain to ssp.sh, as it’s shorter :)

How Did I Manage to Continue to This Day?

[[Writing is hard]] as anyone will tell who does it. So why do I torture myself to do it to this day? Even made it my full time work, as I’m currently self-employed and work as a full time author.

The answer is not straightforward, but to say the truth, I still love it to this day. Writing words is my canvas as an artist, where I can let out my thoughts, be creative, bring something complex into simple terms. Into something that anyone might want to read.

How Has Blogging Changed over the Last 10 Years?

During my start, where I created a WordPress personal blog website, to today, there have been different evolutions, but overall, not much has changed in terms of personal blogs.

These are still the same, except that we changed the technology a couple of times, from using Flash websites to hand writing PHP/HTML/MySQL to using WordPress to Medium and [[Static Site Generators (SSG)]] to Substack today, the main change is social media. Before, personal blogs had more authority. Everyone was linking to other blogs, currently a couple of social media tech giants have the monopoly and you almost need to share there to be discovered.

When I started, I used Twitter and LinkedIn already too, but “the game” of distribution has changed. But again, the sole purpose of personal blogs is the same.

Today with AI we are even in a new era, with all the “AI Slop” generated and shared all over the place. I believe, and see it as my work as a professional writer, that the craft of writing, and [[Writing Manually]], gets even more important.

Writing is communication, and we can’t communicate through a filter, which at the moment many are doing with converting bullets into prose and the reader summarizes from prose to bullet points - watering down the actual points and wording the original author has made. To the point where most people, me included, [[I’d rather Read the Prompt|would rather read the prompt]].

Blogs vs. Second Brain Notes

One approach that I like to push, and many are doing locally with [[Obsidian]], is connected note taking. I shared my Obsidian notes that are worth sharing on my public second brain (find my process at [[Public Second Brain with Quartz]] of adding #publish and it will be on my site, no conversion needed, the code and utilities are shared on GitHub, too)

Bringing back connected personal notes, but also internally on your website - using synergies between your blog and second brain. The way I think as of now about [[Sharing as Second Brain Note vs a Blog Post]]:

The second brain helps me to share whatever is in my mind, and the blog helps me to refine. Notes compound and always evolving. Blog posts capture a moment in time.

There’s also the difference between long-term, always updated, and [[compounding]] notes vs. the one time distilled blog article. They work so well together. As you might notice, most of my links in this article, with much more information, are long-term notes that I’m collecting and refining over the years, linked to my second brain.

This way, I can bring all notes into one storyline, the way I’m currently thinking, sharing it in the form of a blog, as this one, while continually updating the related notes all linked here on long-term strategy for blogging, and with its different [[Type of Notes]].

Connects Knowledge: Helping Learning the Same way as Our Brain Does

I’m thinking of Designing Data-Intensive Applications by Martin Kleppmann), where he added maps to his book, that correlated similar terms:

I see the connected, interactive graph on my second brain, and on my book (just recently added) the same way. It helps learning.

It’s proven that we learn much better if we can associate to an existing term or something we know, versus something new that is orphaned in our brain, without a connection and synapse to another thought (or note in our case). It’s hard to remember and learn from it.

In a [[Second Brain]] and [[Digital Garden]] approach, you connect every note at least to one existing term. I also like to add its origin so I always know where it came from. More on My Obsidian Note-Taking Workflow if that interests you more.

For example the below note about [[Functional Data Engineering]] (← click here to see the graph and backlinks in action) shows how besides the written text, you can glance connected notes through the interactive graph or through backlinks.
![[img_index.en_1772811747298.webp]]

We can visually see things that are otherwise almost impossible to grasp or see. Like a map of a city can convey information density that no chat or explanation can do by explaining to someone on the phone or in written text. It’s the same with the graph.

And the best part, it’s additional, so you don’t need to look at it at all. But most helpful when you want to learn or might not know the space that well yet, you can see a term or connection you know, and immediately connect your brain, that these belong together, probably remember forever, or much longer.

E.g. in the above example, we might see that functional data engineering is linked to clarity, and to [[No Less Code vs Code|Code Is Still the Best Abstraction]], which might be non-obvious, but really helpful to know.

Again, linked notes are the best way to organize knowledge, especially optimized for learning. Knowledge doesn’t grow linearly. It expands as a network over different seasons. I write more about that phenomenon and continue to update at [[Future of Blogging]].

The Process - And the Difference between Long-Term and Short-Term

My process is essentially:

1. idea occurred by reading a book, listening to a podcast, talking to someone or else
2. writing it down in my private second brain
3. connect to existing notes, refine the note, add new thoughts and notes
4. eventually add #publish and publish online on my public second brain
5. eventually distill that note with many others and write a blog post about a specific and related topic of that note
6. continue updating and refining the note
7. eventually writing another blog that relates to it, using that improved note
8. using the note for my book topics
9. refining note

I think you get the gist. The initial one liner, the note that just existed based on a real insight, is the actual most important information of the whole process, in my opinion.

Not to say that the blog articles are not, but they both need each other. With writing the blog, I distill and connect multiple notes at a current timestamp into a frozen article. During that process, notes also get updated. And while sharing the blog post, I get a lot of feedback, [[Feedback Loop]], which I then instead of adding to the blog, which I can’t as it’s a snapshot in time, add to the existing note.

So feedback is actively processed, and massaged into my private or public second brain, improving my overall approach. I think this process of connected notes from private, to public note to blog and continued note, is even more powerful than Niklas Luhmann’s [[Zettelkasten]], that revolutionized the [[Smart Note Taking]] approach based on zettels, small unique ideas that he connected to others.

[!info] No Research needed with this process

Related is also that through this process, [[Why I don’t Research|I don’t need to research]] in a classical sense for topics. As my life and insights come steadily in, and get massaged and integrated like a slow flowing river, all organically.

The Form of Writing: long Form or short

My writing usually tends to be very long-form. Because I take lots of notes that I try to connect and write in an interesting way, I tend to get very long. Same as this writing already is, but I still have 3000+ notes collected to go.

Also, long-form writing evokes a deeper relationship and trust that is hard to captivate with a couple of words. That’s also why a [[Reading Books for a Happy Life|book]] can connect you to the author like no other medium can.

It’s also a question of long term game and writing for it to stay relevant for many years to come, or just capturing a quick trend and harshly (fast?) putting out a blog. These are totally different forms of content, and strategies. The latter usually also used on social media, to create big attention to go viral for a short term.

But what I always ask myself, what’s the gain from it? Ultimately, likes and followers are just a [[Vanity Metric]], and to me at least, don’t count as much as a real human reading these words. Not leaving a like or comment, but just having made a connection or an impact on someone in another part of the world I don’t know (yet? I’m always happy to get introduction emails from my readers! :)). Or just inspiring or making you think about something related, or just learning something new.

That’s at least my main goal. There’s no hidden goal or message behind my writing. Obviously if I write for my clients, it’s a bit different, as I want them to succeed, whereas I write for myself, I just want to let out my thoughts. But what I learned over the years is that [[Writing from The heart]], being genuine, is also helping for work related topics, as at the end of the day, it’s still a human being reading it - and therefore the same principle applies as if it were just a random blog post.

My Writing Process

I’d like now to switch gears a bit as we went through the differentiation of compounding, refined notes and in-time blog posts, and talk about my writing process that I have mastered or improved over the years.

This is my unique writing approach and even more so, note-taking approach. This is not how you have to do it, and probably won’t work for you. As I learned on the How I Write (Podcast) by David Perell, each author has a totally different approach. And this here is mine.

But here I want to share a little bit more about my strategies, my approach to writing, and my tips and tricks I have learned and noted down over the years.

I Spend Many Hours, Weeks and Months on Single Blog Posts

But even before that, a quick prefix: I spend many hours, weeks, sometimes months on a single blog post. Why Are We Here on Earth? for example, I wrote over the course of two years - but if you include my note taking, it sometimes is obviously even longer, because some notes of mine are ten years and older.

There’s also the longer I work on it, the more learning, and sometimes struggles I can put into a piece, which helps the piece to not get outdated the next months too. Something that I’m grappling with over months and years most probably won’t be gone tomorrow.

This is one reason why I don’t like to write too much about AI at the current pace, everything I write, and would spend lots of hours might be outdated the moment I publish.

That’s why my approach is just collecting and refining my thoughts on a second brain note, for example in this case on [[Will AI replace Humans]] and related notes - which made it already on the frontpage of Hackernews - but I’m sure at some point I will take that note and all its relevant related notes, and will distill into a single blog post. But the time hasn’t come yet, as so much is changing.

But it’s not that I don’t do it at all, sometimes I will write something quick, something maybe less long-term, but usually, it’s just less fun for me to write, and maybe less challenging? Although, some topics and articles that I have slept over too long just poured out of me in one go, no sophisticated linking in my second brain or other approach, just a blank page and writing it down. But usually these are topics that I have read extensively about, I’m discussing with people or are just dear to my heart, that my subconscious is just working on it until it’s telling me it’s ready, and then I must not miss the opportunity and just write it down.

A little similar to this piece and topic. It’s so dear to my heart, and something I wanted to write for so long, that I have never done it, and now with the “write that blog” interview, it triggered so many questions that I just went on and wrote until now in one flow. No breaks, just free flow and combining different notes I have in my Obsidian vault.

This is how my vault and process looks right now, with:

1. The current note
2. Is the long long outline (you can’t even see half of it),
3. Is related notes through smart connections
4. Is the initial write that blog interview I answered
5. Is another connected note I have written just above about
6. These are more related notes
7. And you see word counts on the lower right and Vim Motions I write the article in

[!example] Don’t focus too much on the numbers, but on writing
With social media, you could focus too much on [[Vanity Metric]], and how many likes you get on social. But I try not to give too much about it, though it’s still needed. I wrote more about Well Being in Times of Algorithms, my personal essay towards a better World Wide Web, and how well-being is connected to social media.

Ultimate Goal: Good Storytelling

If I had to summarize my writing process, or the goal of it, then it crystallized to me lately that the goal is to have the ultimate storytelling. I want to write about a topic that has an intro that catches the attention, then has a great body, and finishes with a hook and ties everything together.

Storytelling like in the movies, it’s true for writing too, where they have the main act, second act, the villain etc. But with the difference, you can’t use fancy show effects, you are left with simple words.

[!abstract] What is Good Storytelling?

This obviously is very personal, and differs from person to person. To me, most of it boils down to the art of leaving things out, which I am getting much better at over time. And I think that is really what storytelling is all about.

That’s why it’s very important to use what you have as a writer. In writing a blog like this, one of the most important and one I like to use most, it’s the length of a paragraph, making them look good, break at the right timing when re-reading. End on a high, start the next that connects but with a new insight. The paragraphs should be of different lengths, they should be interesting, and change over time.

Mix it up with images, with some quotes or what I like a lot, [[Admonition (Call-outs)|callouts]]. The reason why I like callouts is I can add an additional story, a side note in a way to not distract from the main storyline, but I can serve some readers who like some behind the scenes or more information. Plus they look beautiful in my eyes, adding different colors to the blog post as each of my different types of callout has a different color. It makes it more interesting aesthetically, and that also helps to want to read something more in my opinion, the aesthetic can help big time.

Leave with a Spark✨. Making it Interesting

Besides the ultimate goal of having a good storyline, having a common thread, a nice reading flow and outline is key to keep you, the reader, engaged. I like to jump a bit around.

Not only cutting some topics short, moving on to something else, maybe coming back, maybe not, leaving the reader in the blank, making it more interesting. The key of good writing is leaving out what needs to be left out. It’s an artform, because I could ramble forever on this topic, as I’m super passionate about it - as you might have noticed - but I need to always keep in mind to not bore you. To give you new insights.

That’s why I’m switching now to making it interesting. Besides switching from topic to topic, I also like to go very deep in a topic, and then zooming out very high-level, only to go very deep again in the next sentence.

Zooming in and out helps the reader to not lose the overview, but also learning something new. I usually don’t spend too much time in the middle “zooming level”, this section is boring to me as it’s too vague (not detailed and concrete, and not guiding with not enough overview).

Switching all the time might feel a little like a rollercoaster, but rollercoasters are fun, so do I envision my articles. Writing and its process boils down to me to:

[[Writing from The heart]] is the best. True, honest, and genuine human-to-human communication.

Writing from Curiosity

My best writing comes from my own curiosity. I want to answer a question for myself, even better if I don’t know the answer beforehand. Not knowing where I’m heading to.

I usually set a title, and then go with the flow, see where it leads me—these are the best writings of mine. If I have to write about a certain topic, or outline, it couldn’t be more boring, and that’s usually reflected in my writing too.

The exception is if I know the space very well, I can write a leadership thought piece, bringing together 20 years into one blog. The challenge is again in nailing the storytelling part to make twenty years coherent from start to end.

My Writing Style

Finding your writing style and writing voice is something that was very hard for me. But I think is key to become a writer, especially a professional writer.

It takes time. What helped me for sure, was to read many books, finding my favorite authors and identify their writing style. This helped me to find that I liked Derek Sivers books. Initially, I didn’t know why, until I found more about his writing style, his personality in interviews, read more from him, and analyzed his work.

I found that his minimalistic style, to scrap each unneeded word, straight to the point and providing value while inspiring with different takes that you haven’t heard already a hundred times. He writes genuinely, he also [[Journaling|journals]] 3-4 hours almost daily, thinking and brainstorming a lot in his head and second brain.

That’s also what led me to journal and write in my second brain, like a physician, experiment with different formulas and ideas, to see what comes out. That’s my second brain idea creation. I have the two phases, the idea creation and finishing part. Again, the second brain is where I start with a one-liner, a note from a friend, listing interesting things, linking to existing notes.

Later when distilling into a blog post, or sharing a public second brain note, I will tackle the deeper meaning, the connection with other ideas and areas of my life or things I’m currently learning or have learned a long time ago.

I try not to force it. How many times have I tried to force it, only to go to bed early, and the next day wake up and just have it flow out of my fingers.

Writing is Rewriting

Most of it is also just rewriting. When you write something 3-4 times, when you sleep over it, your subconscious has worked on it while you walk, it always gets better.

It’s a way of personal editing process. It’s also a way of writing style. Jason Fried and Haruki Murakami from Novelist as a Vocation (an amazing book for writers), are constantly re-writing. Sometimes based on feedback of readers, sometimes based on a [[Gut Feeling]].

[!example] A short story from the book Novelist as a Vocation
Haruki Murakami writes in his book that once he lost a manuscript of a chapter. He was devastated, but had no other choice but to rewrite it.

Years later he found the manuscript again and was afraid it would be better than what he had handed in for his book that was already published. But the fear was all wrong, it was so much worse, he writes.

Finding Your Voice, but How?

So, how do you find your voice?

After all the different ways I wrote, I read from other authors, there is no one way, and I can’t tell you how yours will be, other than you start writing, and trusting in the process.

For me, my writing voice I defined as written in personal first person voice. Something I have experienced I can easily explain or write about. But making up a fake story, something they let you do in school, is something I was never good at.

I try to be authentic, friendly and succinct, with the goal of adding value to the readers.
I’m trying to give clues and tools concrete and extremely specific but also leave things out. Because I’m not an academic and I can only make suggestions about what I learned but cannot solve all the problems I write about. I also try not to overly copy others’ ideas, but make them my own through connecting and my own unique life experience. Including hardship and daily struggle and just life.

Key is also that you share in public. Keep writing until you find your voice. From there on, it will be much easier.

How I Found My Writing Voice: Through the English Language

I did live abroad for almost three years in a foreign country, learning Danish, but even more so English. And what happened there is what I would never have predicted.

The more I learned the language of English (I wasn’t fluent before), I started to read more books. I noticed that there are so many more books that weren’t available before, when I only read in German.

Also, I found out that I really liked the English language, the simpler grammar compared to High German as we like to call it in Switzerland. I found that I can express myself much better, more precisely as English has so many words for almost the same meaning, so you can choose and pick one that exactly describes what you want to say. Whereas in German I felt I always need to write a full novel to explain a simple thing very specifically. This might be good for fiction, but not for my technical writing, or also what I write here.

All of a sudden I was reading books all my free time, listening to Tim Ferriss and all his guests on his podcasts, and learning every day. Also, that was the time when I went to the library in Copenhagen and started writing, in English, a secondary language I was just about to learn properly (I had English in school before) and could converse and have small talk. I wrote more about my journey and about Finding My Pathless Path if you are curious to know more about that.

Simple English

As you know now, and might get from my grammatical errors here and there, my English isn’t my mother tongue.

For a long time I saw that as a disadvantage, but lately I figured that it might even be a strength of my writing. With my somewhat limited English vocabulary and language skills, leading to my articles and writing being much simpler English.

And one thing I learned over the years, the easier you can explain complex topics, and make it approachable, the easier for my reader to follow along. Also it makes the writer more approachable, less “snobbish” maybe?

And this is an advantage. My writing is much more approachable this way. It adds a natural constraint to my writing and makes my writing process potentially easier, that I am not even aware of during the writing, but helps me in a certain way I do write.

The Writer’s Toolkit: And the Tools I Use

Let’s come to the last bigger part of this already long article, the tools and methods I use.

My main tool is writing in an open format, that is just [[Markdown]] and then using [[Obsidian]] as the editor to connect these simple notes together in a meaningful way.

The Different Modes of My Writings with ‘Vim Motions’

Apart from that, the way I write is with something called [[Vim Language (and Motions)|Vim Motions]]. I have written extensively about it, and you might think these matter not so much.

I hope, if you write online, or program for a living, that you learned touch typing at some point in your life. If you have, you’d agree that it tremendously helped you with everything working on the computer, right? Not needing to see each key before you press.

Vim motions go a step further, essentially making each key on your keyboard a tool. In the default mode, when opening vim, each key press is doing a function. E.g. g is for jumping around (gg is jumping to the top of the document, ctrl + o jumps back where you left before. G jumps to the end of the document. $ jumps to the end of a line. And so on, I could go on forever, but if you want to actually write something, you’d need to switch to “insert-mode” with i (there’s also a for append or o), but you have different modes. I wrote in [[Four Modes of Writing]] I have four modes with vim motions at all times:

1. NORMAL mode: jumping around, reading, learning
2. INSERT mode: writing, thinking, making connections
3. VISUAL mode: copying, highlighting, format, designing
4. COMMAND mode: automate, fix, macros.

These vim motions, which are different from [[vim]] or [[Neovim]], the editor, are also available in Obsidian and almost any editor you know. Even Gmail has shortcuts like j to go down or k to go up, two common ways of navigation in vim motions. Even more, vim has a language, the vim language. This is super helpful as you don’t need to memorize 1000s of commands by heart, but can combine them. Almost like Streetfighter where you can do a combo.

Besides vim motions, which I write much more on Why Vim Is More than Just an Editor, you can edit at the precision of a surgeon. If interested, also check out my video on [[Vim Motions for Writers]], where I made a timelapse of how that looks:

[!tip] A Trick you can use: Writing within the App vs. a Note app

Sometimes it’s hard to write in your notes app or offline, but if you send an email to a friend, or write in a LinkedIn post or in your blog editor, the pressure is on. You know it’s going to go live, or it’s for a certain friend. This can help you unblock [[Writing is Hard|writer’s block]] or produce better quality.

The Medium

Different mediums to publish, to write on, to take notes are out there, when to use which?

The Medium to Publish

Which medium, which platform do you use? Nowadays you have many options, Substack, Medium, Ghost or other [[Open Subscription Platforms]].

I would always recommend having your own domain. If you like web design and tinkering a bit, even more now with [[Vibe Code Agents|AI Agents Tools]], you should start with a [[Static Site Generators (SSG)]].

These allow you to use Markdown as the format, and owning your content, not losing backlinks when switching platform, and building the domain ranking over time, leading to more authority when searching for a job, or your own business or side projects. Or also just a hobby where you share learnings and topics of interest to you.

[!example] Medium in Taking Notes
I wrote much more on [[Digital vs Paper]], where I explain that I use both - and also share examples of how that process from physical paper notes to digital notes can look like.

Tools: Laptops, (distraction-free) Typewriter, Unitaskers!

An important part is also to make it fun! I do that with different devices. Recently I even used the old typewriter of my grandfather. It showed me the power of [[Uni-taskers]].

The typewriter can only write, not like a laptop where you can surf the internet, or play games or get distracted by social media. Just typing ahead. So refreshing.

That’s where I went down the rabbit hole of [[Distract-Free Typewriter|Distraction-Free Typewriter]] and bought myself a small Micro Journal. A digital device solely for typing. Obviously it can connect to the internet and could do much more - as I installed Linux on it - but the resources are so limited that already running [[Neovim]] that I remodeled to a Wordprocessor struggles to open. So there’s no danger of doing anything else.

Also because of the limited screen real-estate, it lets you really focus on the writing, and less so editing an article. So it’s really a joy to use to exercise my [[Creative Writing]] vein. Just for the joy of writing.

The Future-Proof Format: Markdown

As mentioned already, [[Markdown]] is the format of choice for me. Especially after I was trapped in Microsoft OneNote and its proprietary format, I couldn’t get my own notes out of it. It was super key to have something that will surpass the test of time. And there’s nothing more than [[Plain Text Files]] with Markdown.

I gave a full talk about this topic at Knowledge Management in the Digital Age: From Zettelkasten to Startup Owner, check that out if you want to know more why Markdown, its advantages over [[Rich Text]], and how you can build a note taking setup that works with Obsidian - and even setting the foundation for a solo business like mine.

Markdown has many more advantages. The format has been proven to be the best for agents. I can easily share my public second brain in Markdown with [[Quartz - Publish Obsidian Vault|Quartz]] with no conversion or manually copying notes between rich-text editors and website.

Markdown has all the advantages of simply writing with minimal formatting sugar. It has the advantage that the formatting lives as part of the text, which makes copy pasting not lose all the links or bold/italic etc, which we put for a reason.

Markdown is declarative, meaning you can automate things, you can have the same text, but different engines to present. E.g. I use [[HackMD]] for collaboration (Google Docs for Markdown), and I use Markdown to publish on my website. It’s the same file, the same format, there is no conversion needed.

Compare this to your typical Google Docs, WordPress, Webflow, or other [[Open Subscription Platforms]] such as Substack and Medium. These tend to enforce constraints, you need to always copy your text back and forth, creating copies of your text, potentially losing important formats.

The other big advantage, Markdown files are just [[Plain Text Files|Plaintext Files]], meaning we own the files - no big tech or company can forbid access for us or take them away, they work offline when we don’t have internet, and they are super fast as it’s just tiny files that are locally stored, no round trips to the server.

[!info] My Note-Taking Path from forgetting everything to Obsidian with Vim and Quartz
My path so far with note-taking:

1. Forgetting everything
2. Taking scattered and very detailed notes on multiple devices, apps, and paper
3. Improving during my studies with OneNote, where notes related to work or study go into separate notebooks (no notes for personal notes yet).
4. Starting to create a personal notebook for travels, personal research, etc.
5. There is still a lot of confusion about:
   1. where to store my notes
   2. changing of the folder structure
   3. finding older notes is complex and rarely happened
6. Switching to [[Obsidian]] with a new open format and a different spirit and capabilities.
7. Starting my [[Second Brain]]
   1. Constantly updating my long-time wealth of personal knowledge by adding notes about my health, journals, cooking, books I read, and everything related to my life.
   2. I Started to connect notes and sophisticate my system in a way that I confidentially find it later down my life span, the moment I need it.
8. Start using [[Vim]] and, more importantly, its [[Vim Language (and Motions)|motions]] for fast and effortless note-taking.
9. Sharing them publicly with [[Quartz - Publish Obsidian Vault|Quartz]].

Find the full break down on My Obsidian Note-Taking Workflow.

Wrapping up

I wanted to write much more about the art of writing, how to [[Writing|write]], [[How to Write Well]], and generally more the technique and art of writing. But as this article is already very long, I will save that for another long, distilled blog post in the future. You can follow the above links already, where I wrote a lot, but not in this distilled blog format.

Maybe one day I will write a book about it, I have so much more to say and tell 🙂. What do you think, would you read it? 🙃

Now I want to leave you with some unexpected impacts that writing had on me, and how to get started with blogging:

• My articles and website focus on data engineering, but my most successful posts (in views and virality) are topics about [[Obsidian]], [[Vim Motions for Writers|vim]], and philosophy
• That was a surprise — but now I get it: these were where I had something on my heart, something I poured many years into and put into a single writing
• Career impact: got higher salaries because I was considered known “world-wide” through my blog
• People knew my writing, and typically tend to like you as you give the writing for free
• When I started my own company, I basically didn’t have to sell — being online for so long, people know my writing, my principles, and even my life through my [[Second Brain]]
• The total set of articles is what matters: when people come back to you, that’s the [[compounding]] effect

[!question] The Elephant in the Room: AI
You might ask, but what about AI content, isn’t that a reason to not start to write? I’d say no for the same reason writing was good before AI and before books, and before time.
It’s good for yourself, to calm down, let out all your thoughts. Distill, learn and also remember things that are very important to you.

[[Writing Manually]], as I like to call it, is also my joy of writing - if you enjoy it, I enjoy it. It’s hard, it’s a challenge, it’s not easy. I get great satisfaction. Like chess, computers are much better, but we still play chess. And also for the love of words and communication, writing is communication. And to prevent more “AI Slop” from being created.

Read much more on [[Will AI replace Humans]], where I share my latest on that topic.

I have so much more to say, which a lot of it is in my second brain, so feel free just to browse and explore my second brain. Use the backlinks and the graph to explore more. I even added a semantic search, so you can find hidden connections on my public brain, on topics that might interest you, or on this very topic you just read.

A good book recommendation that is related (I share more on Book Recommendations and Notes), that helped me a ton, which I also wrote about finding mine, is The Pathless Path by Paul Millerd. This is the Tim Ferriss 4-Hour Workweek book, updated for today, and it inspired me to take the step of going full-time as a freelance technical writer and making writing my job.

What if you want to get started yourself? Read my interview on “Write that Blog” where I share more suggestions about starting your own blog. But generally, [[writing is hard]], just get started.

Have a note taking app (one only, based on an open format, available offline too), and write things down. Save important moments in your life, blessings of people telling you, insights from books you read, etc.

Follow the mantra of [[Learn in Public|Learning in Public]]. Use the feedback loop, let’s share and learn together. And know that over time, all your notes and personal knowledge will compound, like money does if you invest it cleverly.

[!note] Maybe the easiest way to get started: just an email converted to a blog

Hey World has this feature integrated into their email — you can write an email as you would normally, just a different recipient, and if sent, it will be online as a normal blog post. A very easy way to get started.

I’m leaving some links to tips and tricks for unblocking and finding a good rhythm.

• [[Coffee Break Rhythm]]: Using location pressure as a productivity forcing function. For me, in summer, it’s moving every 1.5-2 hours from coffee shop to coffee shop, creating a pressure to finish up before leaving. It tricks the brain into not thinking: “Oh, I have still all day long time” and then procrastinate, the opposite of [[Productive Procrastination]], where we let go on purpose to get some insights we wouldn’t have gotten otherwise.
• Use the doubts, the signs that you can’t write today. Take a walk instead, best would be to walk everyday. I call it the [[Productive Procrastination]]. Many see it as a bad thing, but it’s unavoidable, and me, and also others, believe it’s our body, gut telling us something. E.g. Tim Urban from Wait But Why is also big on procrastination. He says the same. He hates to procrastinate sometimes, but that’s how his brain works, and where he gets some insights he wouldn’t have gotten without.
• [[Ultradian Rhythm]]: Know that the first 10 minutes of a 90-minute deep work block are always going to be hard. That’s okay. Remind yourself of this when starting is hard.
• Embrace the New Luxury of Boredom: I feel we are at a turning point. We, the people, want to own things, want distraction-free experiences, and above all, want tools that benefit us, not the pockets of large companies. There are more stories that people use old iPods for music, buying the music. Typing on a typewriter solely for writing (like I did distraction-free typewriter), leaving the cloud, or just using local first products like Obsidian, DuckDB. Devices that are uni-taskers, doing one thing well.

Now it’s time to see what Git-like tools for data are out there, and how they actually work in practice. Part 2 dives into the tools and implementations. We’ll examine LakeFS, Dolt, Nessie, MotherDuck, Bauplan, and more, exploring how they work under the hood. Each tool takes a different approach to the same fundamental challenge: enabling Git-like workflows without copying petabytes of data.

The key insight from Part 1 was that all these tools separate metadata from data, using techniques like copy-on-write and pointer manipulation. But the devil is in the details. Some tools version entire data lakes, others focus on databases. Some support full merge workflows, others prioritize instant forking. Understanding these trade-offs will help you choose the right solution for your stack.

There will be gaps, and implementations are changing fast, so take it with a grain of salt. But this should give you a good overview of what’s out there, and help you invest more time in the ones that fit your use case best.

Let’s get into it.

Git-like Tools: Overview

There are many tools out there, some of which have been used for years, and others are rather new. We compare them and see what each of them has to offer.

Comparison Overview

The overview below serves as a summary. We will go into more detail, with each tool getting one short chapter with a showcase of features and application use cases.

It’s by no means complete, but it shows the most dominant players.

Further analysis of the OSS ecosystem of git for data tools and their GitHub activity tells us how healthy the repos are, as of February 2026:

And community responsiveness based on ossinsight.io, latest available month - click on link below to get a deeper insight in each repository:

Note: All data from GitHub API, Feb 2026. Github Activity Chart. See also GitHub Star History

Dolt stands out with the fastest PR merge times (~30 min median). lakeFS leads in total PR creators (178), reflecting a broad contributor base. Nessie’s near-instant issue response reflects automated triage.

[!note] How Do They Work?

While Git versions code through file snapshots and diffs, data tools must handle actual data, if possible, without copying entire datasets. Each tool solves this challenge differently, but they share a common approach: separating metadata from data.

Instead of duplicating data, they track pointers and references, enabling instant branching/cloning and zero-copy operations.

What usually happens without tools like this Testing in Production

Find more insight about the architecture and behind the scenes in Part 1, Branch, Test, Deploy: A Git-Inspired Approach for Data.

Git-like Tools: Break down

Let’s get started with the tools and see their features and how they work, categorized into three categories: data lake based, transactional and relational databases, and analytical databases.

Data Lake Versioning (Object Storage)

Data lake versioned tools sit between the compute engine and the object storage (S3, GCS, Azure Blob), leaving you free to query with whatever engine you prefer: Trino, Spark, DuckDB, etc.

LakeFS

LakeFS is one of the first tools to bring git-like versioning to object-storage-based data lakes. Its core approach is a metadata layer over object storage with immutable data and logical-to-physical address mapping on top of an object store such as a data lake, hence “lake” as part of the name.

It segregates data data/ with random physical addresses from its metadata _lakefs/, which includes range files, meta-range files, and commit information.

When you upload allstar_games_stats.csv to branch main, lakeFS generates a random physical address like s3://bucket/data/gp0n1l7d77pn0cke6jjg/cg6p50nd77pn0cke6jk0. This ensures immutability and files are never overwritten.

LakeFS operates as an S3-compatible gateway, intercepting read/write operations and managing versioning transparently. Applications interact with it like normal object storage while getting full Git semantics underneath.

The system implements a layered architecture:

1. Graveler: Core versioning engine managing branches, commits, and merges
2. Storage Adapter: Interfaces with S3/GCS/Azure
3. Hooks: Pre-merge and post-commit validation

Creating a branch from the CLI is as simple as this:

1

lakectl branch create lakefs://quickstart/denmark-lakes --source lakefs://quickstart/main

The UI supports creating pull requests, or branches, literally like GitHub but for data.

Check out their GitHub repo, documentation, or a practical example of Implementing a Write-Audit-Publish (WAP) Pattern for much more information.

Nessie

Nessie came out of Dremio and is another early adopter that has been doing this for a long time. Its core approach is a transactional catalog with Git-like versioning for Apache Iceberg and Delta Lake tables.

Rather than versioning data files, Nessie versions the catalog metadata, the registry of tables and their locations.

This separation enables zero-copy branching where branches share table metadata pointers, multi-table transactions with atomic commits across multiple tables, and Git semantics such as branch, tag, merge, and cherry-pick operations.

Nessie leverages the immutability of modern table formats with Iceberg:

1. Iceberg snapshots are immutable: Each table change creates new metadata.
2. Nessie tracks which snapshot each branch points to.
3. Branching copies pointers, not data or metadata files.
4. Merging updates pointers to replay changes from source to target.

Example workflow:

1
2
3
4
5
6
7
8
9

# Create branch
catalog.create_branch('experiment', 'main')

# Modify table on experiment branch
spark.sql("INSERT INTO catalog.experiment.orders VALUES (...)")
# This creates new Iceberg snapshot, Nessie updates experiment pointer

# Main branch unchanged - still points to original snapshot
spark.sql("SELECT * FROM catalog.main.orders")  # Original data

Nessie runs as a REST service with pluggable backends including metadata storage such as PostgreSQL, DynamoDB, or RocksDB, data lake integration that works with any Iceberg-compatible engine (Spark, Trino, Dremio), and version control with a Git-like commit graph with branches and tags.

Nessie doesn’t touch your data files. It’s a lightweight coordination layer that brings Git semantics to your lakehouse by versioning the catalog. This makes it complementary to tools like lakeFS (which versions data) and ideal for multi-table transactional workflows. Read more on GitHub.

Bauplan

Similar to LakeFS, Bauplan calls itself the programmable data lake and is a code-native platform for versioned pipelines, built on Apache Iceberg and initially optimized for ML. It’s not open source. Bauplan is built on a Python-first serverless lakehouse and is rather new.

Bauplan treats your data lake as a Git repository where:

• Data branches are first-class citizens, not just pipeline configs.
• Every pipeline execution is a commit with full lineage.
• All tables use Apache Iceberg format (Delta Lake compatible).

Creating an isolated branch with new snapshots of Iceberg tables from the CLI is as simple as this:

1

client.create_branch('experiment')  # Instant, zero data copying

It supports merging verified using Alloy model checking:

1

client.merge_branch(source='experiment', target='main')

The way it works is that it integrates a commit’s changes into another branch and uses Alloy, a lightweight model checker, to stress-test the core logic behind merging (also used for checking branching and commits).

The merge operation tries to detect conflicts at the table level, performs three-way merges for compatible changes, and creates merge commits preserving lineage. Find more info on Git-for-Data Semantics: Safe Branching & Merging at Scale or their implementation of the WAP pattern.

Bauplan brings Git’s full semantic model with branch, merge, commit, and revert to lakehouse data while maintaining compatibility with standard Iceberg tables accessible from MotherDuck, Snowflake, Databricks, or Trino.

[!tip] Software Modeling with Alloy

I haven’t heard of Alloy before, but it’s used not to model data, but for software modeling. It’s used for a wide range of applications from finding holes in security mechanisms to designing telephone switching networks. And now for git for data with Bauplan.

[!note] New Whitepaper Out

After this article was written, Bauplan released a new whitepaper on Building a Correct-by-Design Lakehouse that researches around pipeline boundaries with Git-like data versioning for review and reproducibility, and transactional runs that guarantee pipeline-level atomicity.

Transactional and OLTP Databases

These are row-oriented, ACID-compliant databases where Git-like versioning applies mostly to application data where we need to keep user records, orders, and schemas.

Supabase, Neon and Dolt are interesting because these are not data lakes, not based on object storage, and not analytical databases, but relational databases.

Supabase

Supabase’s core approach is full instance branching. Each branch is a completely isolated Postgres database with the entire Supabase stack (Auth, Storage, Realtime, Edge Functions).

Supabase branches create separate environments that spin off from your main project, allowing you to test changes like new configurations, database schemas, or features without affecting production.

It works by creating a Git branch and opening a pull request. Supabase automatically launches a Preview Branch and runs migrations from the repository’s migrations directory. Each branch gets a dedicated Postgres instance with a unique connection string and APIs, isolating them from production and other branches.

Creating a branch via GitHub integration:

1
2
3
4

# Automatic with GitHub integration enabled
git checkout -b feature/new-reports
git push origin feature/new-reports
# Supabase automatically creates preview branch when PR is opened

Or via the CLI:

1

supabase branches create feature-branch --project-ref your-project

When merging, migrations in the repository’s migrations folder run incrementally on each commit, allowing you to verify schema changes on existing seed data. When you merge the PR, those migrations automatically apply to production.

As each branch is a new Postgres instance created from scratch, the approach is conceptually simple but requires branches to be seeded (manually populated with test data since production data isn’t copied) with data since they start empty. Each branch incurs its own compute and storage costs. Read more on Branching Supabase Docs.

Ideal for full-stack development where you need the entire backend stack (database + auth + storage + functions) to test features end-to-end.

Neon

Neon is a serverless Postgres platform (now part of Databricks) whose core approach is copy-on-write storage-level branching. Unlike Supabase which spins up a full new instance, Neon branches at the storage layer, making them instant regardless of database size and including the actual data.

Each branch is a new timeline in Neon’s custom storage engine. No data is physically copied. The branch simply starts from a pointer to the parent’s state at a specific LSN (log sequence number). Pages only diverge when writes happen, so you’re billed only for the delta.

1
2
3
4
5

# Create a branch from the CLI
neon branches create --name feature/user-auth

# Branch from a specific point in time
neon branches create --name recovery --parent 2025-01-15T10:00:00Z

Neon also supports snapshots (named, immutable point-in-time saves, like git tags) and rollback via finalize_restore: true, which restores a snapshot onto the active branch in-place while preserving the stable connection string. There’s no reconfiguration needed. For safe experimentation, finalize_restore: false creates a temporary preview branch instead.

The key limitation: Neon has no merge support. Branches diverge but can’t be reconciled automatically. Changes are applied back to production using standard migration tools.

Ideal for database-focused workflows where you want instant, full-data branches with production-like data out of the box, and don’t need the full backend stack.

Dolt: Git + MySQL

Dolt is a SQL database that you can fork, clone, branch, merge, push, and pull just like a Git repository. It’s a MySQL-compatible database and is fully open-source. Dolt’s core approach is a SQL database where every row is versioned, combining Git’s commit graph with MySQL’s query interface.

Dolt stores data in a content-addressed graph using Prolly Trees, a novel data structure that enables cell-level version history, efficient structural sharing between versions, and fast diffs and merges.

Every database operation can be committed with:

1
2

INSERT INTO employees VALUES (1, 'Alice', 50000);
SELECT DOLT_COMMIT('-am', 'Add Alice to payroll');

The commit creates a snapshot of the entire database state at that moment, stored in the commit graph just like Git. Unlike traditional databases, you can diff any two versions:

1
2
3
4
5

-- See what changed between commits
SELECT * FROM DOLT_DIFF('main', 'feature-branch', 'employees');

-- Show cell-level changes
SELECT * FROM DOLT_COMMIT_DIFF_employees WHERE from_commit='abc123' AND to_commit='def456';

This enables cell-level audit trails with diffs showing which rows were added/deleted/modified, which cells changed with their before/after values, and who made the change via commit metadata.

Dolt implements Git commands almost literally. You can run dolt with any of these commands: branch feature-123, checkout feature-123, add ., commit -m "Add new customers", push origin feature-123, checkout main, merge feature-123.

You can even push/pull to DoltHub (like GitHub for databases) or run Dolt as a MySQL replica for existing applications.

Dolt uses copy-on-write with structural sharing where unchanged rows are shared between branches via pointers, and modified rows create new leaf nodes in the Prolly Tree.

This means cloning isn’t “free” like with lakeFS, but it provides true database semantics with ACID transactions.

There’s much more. Read more on their GitHub.

[!note] Worth noting

DoltgreSQL, the Postgres-compatible version of Dolt, reached Beta in 2025 and is available on Hosted Dolt. If your stack is Postgres-based, DoltgreSQL brings the same Git-like versioning semantics without requiring a MySQL migration.

Analytical Databases & Warehouses

These tools are OLAP-style and analytical-style databases optimized for read-heavy analytical queries.

MotherDuck

MotherDuck, as a cloud data warehouse, implements versioning differently from dedicated Git-for-data tools, prioritizing operational convenience over full version control semantics. With the addition of named snapshots, it gets even closer to Git-like semantics.

It offers two types of snapshots. Automatic snapshots: Created continuously in the background (roughly every minute when no writes are active). These are governed by SNAPSHOT_RETENTION_DAYS. These are configurable up to 90 days on the Business plan, defaulting to 7 days. They provide point-in-time recovery without any manual intervention.

And named snapshots that you create explicitly with CREATE SNAPSHOT. These are not subject to garbage collection as they persist indefinitely, even if the source database is deleted. Think of them as Git tags for your database, a permanent bookmark of a known-good state you can always return to.

The git analogy maps well:

1. CREATE SNAPSHOT → git tag: bookmark a known-good state
2. CREATE DATABASE ... FROM → git checkout -b: isolated environment from a snapshot
3. ALTER DATABASE SET SNAPSHOT TO → git reset --hard: roll back to a previous state
4. UNDROP DATABASE → recovering a deleted branch

Combined with zero-copy cloning and database sharing, this enables practical git-like workflows. While MotherDuck doesn’t support Git-style merging, COPY FROM DATABASE (OVERWRITE) acts as a replace, somewhat like a merge without conflict resolution. Combined with snapshots and zero-copy clones, this gives you a practical branch-modify-promote workflow:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

-- 1. Snapshot production before changes (persists indefinitely)
CREATE SNAPSHOT 'pre_release_v2' OF production;

-- 2. Clone from that named snapshot to an isolated dev database (instant, zero-copy)
CREATE DATABASE dev_branch FROM production (SNAPSHOT_NAME 'pre_release_v2');
-- Or clone from a point in time: (SNAPSHOT_TIME '2026-01-28 08:00:00')

-- 3. Make and validate changes on dev_branch
-- ... run transforms, test queries ...

-- 4. Promote: overwrite production with dev_branch (instant, metadata-only)
COPY FROM DATABASE dev_branch (OVERWRITE) TO production;

-- 5. If something goes wrong, restore from snapshot
ALTER DATABASE production SET SNAPSHOT TO (SNAPSHOT_NAME 'pre_release_v2');

This operates purely at the metadata layer and is nearly instantaneous. It’s not a true merge (it’s a full replacement, not a diff-based reconciliation), but for many data workflows where you want to validate changes in isolation before promoting them, it covers the key use case.

[!example] Deep Dive

If you want to know even more about how to use named snapshots and generally rolling back to a certain time, this blog More Control, Less Hassle: Self-Serve Recovery with Point-in-Time Restore goes into more details.

DuckLake

DuckLake is the open lakehouse format that uses a SQL database as its metadata catalog instead of JSON/Avro manifest files. DuckLake is relatively new (with 1.0 around the corner and its first release in May 2025), so you could use other mature open table formats like Apache Iceberg, Delta Lake or Apache Hudi.

But DuckLake has its relevancy for git-like workflows because:

1. Snapshots are Git commits: Every DuckLake change creates a snapshot with author, commit message, and changeset tracking. This is the closest to actual Git semantics in the data lake world.
2. SQL-native metadata: Uses DuckDB/PostgreSQL/MySQL as catalog, so metadata operations are standard SQL transactions. No manifest file scanning or compaction storms like Iceberg.
3. Millions of snapshots: Snapshots are just a few rows in the catalog DB. No need to proactively prune snapshots (a major operational burden with Iceberg).
4. Time travel + change feed: Query any table at any version, track insertions/deletions between versions.

With MotherDuck (fully managed):

1
2
3
4
5

-- Fully managed DuckLake on MotherDuck
CREATE DATABASE my_lake (TYPE DUCKLAKE);

-- Or bring your own S3 bucket
CREATE DATABASE my_lake (TYPE DUCKLAKE, DATA_PATH 's3://my-bucket/lake/');

[!example] DuckLake Example

See valuable examples and DuckLake workflows in DuckLake workshop.

Related Data Engineering Git-like Workflows

Besides storage for data, which is the most important part and at the same time the hardest as we need to deal with state, it’s not the full picture. We have DataOps to handle the full picture.

Data pipelines and their code also need to be deployed on a clone or branch, so how do we do this? One example is orchestration.

Orchestration: Dagster Branch Deployments

If we look at the full picture of the data engineering lifecycle, we need more than just storing data in a git-like manner. To support the full lifecycle, it would be best to run everything in a git-like style to roll back or switch branches. It’s great to see that orchestrator tools like Dagster and others also have this functionality included.

Meaning branching does not only apply to the data, but also to data pipelines, and we can set a run automatically. Dagster is doing that with their cloud solution, integrating GitHub workflows with PRs and actions.

Dagster’s core approach is lightweight staging environments created automatically with every pull request that branch both code and data. Branch deployments deploy your branch on Dagster+ as a separate deployment. This only works if your underlying technology supports cloning. For example, as we’ve seen, one of the above tools that supports cloning will allow Dagster inside the deployment to clone relevant data into that new branch deployment.

On PR creation, it will automatically create a staging environment with a branch, launch jobs to configure the test environment including cloned data(base), and allow parameterized pipelines to test. If the tests pass, you can approve the PR, and it merges and automatically deploys to production with the right CI/CD pipeline.

Orchestrators and other data stack tools depend on cloning support and features such as branching for a true isolated environment. As Nick Schrock noted in the Data Engineering Podcast, this is similar to the challenge with Apache Spark where testing locally is nearly impossible. Branch deployments solve this by branching the entire environment.

This is extremely powerful as it replaces the need to copy data locally or set up complex staging environments. You get a true production-like test environment that’s automatically created and destroyed with your git workflow. Read more on Dagster Branch Deployments.

AI Agents: A Branch for Testing

Lastly, this also works well in the realm of AI agents that help us test based on a branch or snapshot. This is similar to git worktree for small git repos with code where basically each branch is a separate folder and we can work and change different branches simultaneously without breaking any of the other branches or data.

Once we have a working branch with data included in isolation, we can send off an agent autonomously, and let it open a PR to review. This way we have a clear gateway before it goes to production, we can test it on that branch, including its data, and merge when all looks good.

Based on its own fork, we can avoid collisions, instantly roll back or delete a branch and start again, have perfect consistency as data is frozen and locked for the agent to work on, and clean debugging as no other ETL data pipelines interfere.

Conclusion

So where does this leave us? In Part 1, we established that Git for data is fundamentally harder than versioning code because we’re managing state at massive scale. We learned about the efficiency spectrum, from metadata pointers to full copies, and why zero-copy operations matter.

Now, having explored the actual tools and their approaches to git-like workflows (LakeFS, Dolt, Nessie, MotherDuck, and others in production today), we know a little more about how it all works. Each tool makes different trade-offs, but they all solve the same core problem: how do you version data without copying petabytes.

The answer, to me: separate metadata from data. Whether it’s LakeFS’s random physical addresses, Dolt’s Prolly Trees, Nessie’s catalog pointers, MotherDuck’s zero-copy clones, or Neon’s branching feature, they all use clever tricks to make branching instant. Some focus on data lakes, others on databases. Some support full merge workflows, others prioritize instant forking. Your choice depends on your stack:

• LakeFS and Nessie excel at data lake branching with zero-copy efficiency
• Dolt brings true Git semantics to SQL databases
• MotherDuck offers named snapshots and zero-copy clones for cloud data warehousing, with DuckLake adding SQL-native time travel
• Bauplan focuses on versioned pipelines and ML experiment reproducibility
• Neon and Supabase provide branch/fork-based workflows for isolated testing

The ecosystem is still evolving. Maturity varies across tools, with different workarounds to limitations that best fit data in a git-like workflow. Some trade merge capabilities for instant forking. Others require infrastructure changes. The key is picking what fits your workflow and scale.

Start small. You don’t need to instrument your entire stack overnight. Look at your recent production incidents: which pipelines caused them? Those are your highest-risk areas. Add branching there first. Test changes on prod-like data before deploying. Build confidence through small wins, then expand.

We want to bring the same confidence we have with code versioning to the stateful world of data. And with tools like Dagster’s branch deployments and emerging AI agent workflows, we’re seeing Git-like patterns extend beyond just data storage into the full data engineering lifecycle.

Git-like workflows are becoming table stakes. Maybe not today or tomorrow, but with the right tools and changes in workflow we can achieve significantly better change management, testing on production data, fast rollbacks, isolated experiments, and most importantly, peace of mind when deploying changes.

That’s the promise. What’s your experience? Have you tried it? Do you run any of the above in production? I’m curious to hear more.

Appendix

While I was writing this article back in November 2025, Tigris was an interesting database contender with Supabase-like features such as forked buckets and zero clone. But at the time of this publishing, the GitHub repo got archived, and therefore removed from the comparison in this article.

Full article published at MotherDuck.com - written as part of my services

So I built one. A RAG system that runs locally with DuckDB as a vector database, then syncs to MotherDuck for a serverless web app running entirely in the browser via WASM. Think of it like J.A.R.V.I.S¹ for your markdown files: search about a topic, and it shows connected notes up to two hops away, semantically similar content, and hidden connections between ideas that share no direct links.

In this article, I walk through how I built this and how it works, from using DuckDB’s vector extension locally to serving embeddings through MotherDuck’s WASM client. Along the way, you’ll see how data engineering skills can make use of lots of note-markdown files. If you want to dive straight into the code, it’s all on GitHub at Obsidian-note-taking-assistant, and you can try the web app on my public notes at Explore RAG.

For building the web app I used Claude Code and it came together in a few hours using the plan mode. This approach is powerful for any data engineer building pipelines or related work, especially when you have a clear vision of what you want. The big productivity boost wasn’t only the model getting smarter, in my opinion, but something else, more on that in the article.

This is how it looks. Let’s talk about how I built it and some behind the scenes.

Vision & Why I Built This

I have 8963 local notes (according to find . -type f -name '*.md' | wc -l) in my Obsidian vault, some are very long, and there are more images and PDFs connected. Wouldn’t it be nice to have an insight from my own thinking a while back, or some quotes I forgot², or things you didn’t think of?

The requirements that I set myself were to use Obsidian backlinks as these are already curated and well structured as a graph-like organization. I wanted to see notes that are multiple hops away and hard to see without a tool. I wanted to search non-obvious neighbors or similarities and also show me hidden connections that would be interesting, both locally and online. These are especially helpful in the brainstorming and initial phase when starting an article or a note, giving me new ideas on existing notes I have written once in my life.

Examples could look like this:

Show me my notes on Functional Data Engineering that relate to my current article (one or two hops)

Notes that are relevant from my vault. Or related ideas

Highlight any disagreements between the notes

Give me all notes I took on these matters and related, and give me the source note from my Obsidian vault

Such a tool is especially helpful during brainstorming when writing my articles, or when I journal some ideas or when solving a hard problem. All of this should be local, but also available as a web app, so I can share it with you and connect it to my public second brain.

Starting Position

With Obsidian, there are many Obsidian plugins such as Graph Analysis, Obsidian Smart Connections and many more, that let you do similar things. But some require to hook up a public AI provider, don’t work very well anymore, or don’t do exactly what I wanted.

The easiest would be to use Claude Code or any other agents, as it’s just Markdown files, but again, then you give away all your sensitive, potentially insightful notes and thoughts. That’s why I wanted to build an Obsidian knowledge assistant that is trained based on my data. I started with a simple Retrieval-Augmented Generation (RAG) system that uses DuckDB for storing vectors. I used Vector Similarity Search Extension for storing vectors and did a couple of tests with Claude Code.

I shared it online and got helpful feedback to use a specific model bge-m3 and integrated it as much as possible with the help of agents. I added the above requirements that it should use Obsidian native links and train based on my vault.

This was my first round. Building a job that creates chunks and ingests them into DuckDB with the vector extension Vector Similarity Search Extension.

I used two different modes, as the above takes more time to generate embeddings. I could run the BGE-M3 overnight and it was done after ~2 hours, not on all my notes, but on my public notes, which are 584.

Local-First

I started with the local-first approach because I want to be independent, and also I have sensitive or valuable notes that I don’t just want to give away or upload to the cloud.

But there are also other reasons why you might want to use a local model. Some say:

A.I. research done by a cloud service will hallucinate because you have no control over the weights or limits of the LLM. This is why anyone who wants to do A.I. should run their projects locally including Deep Research. Bsky

Additionally, a local model with lots of your own context to research with will be better suited for your use case. It doesn’t mean that it does not hallucinate, but what I find most useful is that suggestions and ideas are based on my own notes, which I sometimes have forgotten, or if new ideas, they are combined based on my research.

Web App

I added a web app that uploads the generated embeddings to MotherDuck and uses DuckDB WASM to serve in the client (web browser), so I could share the findings easily with anyone interested in my second brain notes.

This went really well, and I share all the details at the end of this article, with some lessons learned and how you can do it for yourself too.

Knowledge Assistant: Building a RAG for Data Engineers

Now let’s get to the building part. As initially explained, this article converts data engineering knowledge into a searchable tool. Hopefully finding new insights, related topics, and learning something new.

This is now done on top of my public (mostly) data engineering notes, but we might add code snippets, interesting quotes, etc. To me, all of these might just be text files, and mostly markdown, that’s why this system based on text files is so powerful. We can use it as context to help us more.

The outcome and connected web app looks like this:

What We Built: Retrieval Without the LLM

A Retrieval-Augmented Generation (RAG) system that is trained on our notes that we have (we use Markdown). More specifically: Obsidian Markdown, that has the advantage of links and backlinks that give us additional clues we can use.

RAG in particular is a technique that can provide more accurate results to queries than a generative large language model on its own because RAG uses knowledge external to data already contained in the Large Language Models (LLMs).

So what we built is only the Retrieval and Augmented part. We don’t use an LLM yet, only retrieval of relevant and hidden notes based on a search. Specifically notes, code snippets as parts of notes, and other relevant ideas.

Architecture with Embed Model, MotherDuck and Next.js

First I had to split my notes into separate chunks and connect relevant links.
This is done through an embedding model that converts text into numerical vectors, so we can compare meaning rather than just keywords.

This runs locally and two models can be used: all-MiniLM-L6-v2 (384 dimensions, fast for testing) and BAAI/bge-m3 (1024 dimensions, production quality). This is the top-level Python code in the GitHub repo. It provides a CLI and DuckDB database where we can search semantically, discover hidden notes, or traverse connected notes up to two hops away.

The chunking is markdown-aware: it respects heading boundaries, preserves code blocks intact, and splits on paragraph breaks. Each chunk stays around 512 characters and carries its heading context along. Before embedding, I prepend the note title and section heading to each chunk (e.g., "Title: DuckDB | Section: Installation | actual content...").

This acts as a semantic anchor and noticeably improves retrieval quality.

Disclaimer: I don’t have deep expertise in building RAG systems and semantic search, so this is built on the best of my knowledge and what helps me most in my daily work.

The ingestion pipeline creates these tables with relevant information:

• notes: Note metadata, content, frontmatter
• links: Wikilink graph edges
• chunks: Chunked content for RAG retrieval
• embeddings: 1024-dim vectors (BAAI/bge-m3)
• hyperedges: Multiway relations (tags, folders)
• hyperedge_members: Note membership in hyperedges

The second part is a web app served via a Next.js UI and a MotherDuck WASM client that connects directly to the MotherDuck cloud database from the browser.

This means no database server to set up or maintain. I added a FastAPI service on Railway to serve the BGE-M3 embedding model, which avoids API costs from Hugging Face (and also makes it reliable, since Hugging Face’s inference API kept timing out with the BGE-M3 model).

The architecture uses mostly serverless components:

Semantic search matches meaning, not keywords. When I search for “how to model data in a warehouse,” I want notes about dimensional modeling or dbt transformations to show up, even if they never use those exact words.

The BGE-M3 model converts each chunk into a 1024-dimensional vector, and we rank results by cosine similarity between the query and stored embeddings. Locally, DuckDB’s VSS extension handles this with an HNSW index.

In the web app, MotherDuck’s WASM client doesn’t have VSS, so I compute cosine similarity manually with DuckDB’s list functions. I was surprised how well DuckDB handles this without a dedicated vector database, one file for relational data and vectors together.

The “graph-boosted search” mode multiplies similarity by 1.2x for notes that are also graph-connected. Simple, but it surfaces better results because your link structure encodes intent that embeddings alone miss.

And the hidden connections feature, finding semantically close notes with no direct wikilink, turned out to be the most useful discovery tool.

It found links between notes I’d written months apart and never thought to connect.

Running It on Your Own Vault

As we constantly add and improve our “second brain”, this is very powerful, so we can just rerun the ingestion and we get the update.

This is built on my data, but you can use the provided GitHub repo and run the local make ingest job to run it on your own Obsidian vault or Markdown files. You’ll get the same UI and CLI to ask questions about your notes out of the box.

The results are tailored to our interests, needs, and even notes, as we are the ones who wrote the notes down. Or if you took a lot of highlights via web clippers ReadWise read-it-later, Obsidian Webclipper, also from other authors, but still snippets that you chose to store.

To run it on your own notes, clone the repo, set VAULT_PATH in the .env file to your Obsidian vault (or any folder of Markdown files), and run make ingest.

The ingestion parses all .md files, chunks them, generates embeddings with the BGE-M3 model, and stores everything in a local DuckDB file. From there you have the full CLI with semantic search, backlinks, connections, and hidden link discovery.

If you want the web UI too, sync to MotherDuck with make sync-motherduck and deploy the Next.js app.

The Final Result

The result of this exercise is two parts with sub-components like this:

• Ingestion pipeline: A local job that parses Obsidian markdown, chunks it, and generates embeddings using the BGE-M3 model. Run make ingest and the local DuckDB file is ready to query.
• Web app at explore.ssp.sh, composed of three services:
  • Frontend on Vercel: Next.js app with MotherDuck WASM client running DuckDB queries directly in the browser.
  • Database on MotherDuck: Cloud-hosted DuckDB, synced from local via make sync-motherduck. No server to manage.
  • Embedding microservice on Railway: A FastAPI endpoint that hosts the BGE-M3 model and converts search queries into vectors on demand. The browser sends your search text, gets back a 1024-dim embedding, and uses it to query MotherDuck for similar chunks. This avoids running a ~1.8GB model in the browser and sidesteps Hugging Face API rate limits.

Here you can see backlinks and hops that go over two notes. The hops are interesting as we don’t see this easily on a graph, or it’s harder to showcase. That’s why I added them besides the normal backlinks and outgoing links.

Find hidden connections. Here we see that AT Protocol, the protocol behind social media platform Bluesky and others, is connected to Ducklake. Something I wouldn’t have associated myself:

Now we can compare notes, think why this could be, and what’s the connection and insight we can gain from it. This is exactly why I built this, to get such insights.

[!info] Clickable Links

Each note on explore.ssp.sh has a clickable link to my public brain at ssp.sh/brain/[note-name].

Lessons Learned: AI Agents for Data Engineers

As you probably have noticed, since the Christmas break, the AI hype or enthusiasm around agents got very loud. One reason is that many got a good amount of time to actually test the latest. On the other hand, the models got better, and thirdly these AI companies provided new features such as Skills, cowork, and many more.

I myself also took some time and thought about how we can leverage agents for data engineering, especially Claude Code. But contradicting many who say the models got much better, I think the key to the boost of productivity is a different one. With nao, ChatGPT, Claude, and probably others, we have had AI agents and models already for a while, but most powerful at the current moment are the agents in plan mode. It’s the key to build longer and have us more in the loop.

But what is “Plan Mode” you might ask? The definition:

Claude Plan Mode is a read-only state in Claude Code, an AI coding assistant, that lets it analyze a codebase, ask clarifying questions, and generate detailed implementation plans without making any actual file changes or executing commands, ensuring safety and structure before development begins. It’s activated by cycling modes (often Shift+Tab) and is great for exploring, planning complex changes, and building context, allowing developers to approve the AI’s strategy before actual coding starts. More on What Actually Is Claude Code’s Plan Mode?

With that, it’s amazing what you can build. All the open todos we add to our backlog, we can now quickly build and test or solve, and think through the problem by actually laying out the step-by-step instructions. After it’s built we get a feel for it quickly and can give better feedback on whatever job we have at hand right now.

Still we need to be careful to not just jump into building every little thing, as we could, because spending hours on something that we don’t need is still wasting precious time.

I have experienced it myself often. I get the perception of being super productive, but after a couple of hours, or sometimes days, we actually didn’t achieve what we needed. The idea we thought was cool didn’t go anywhere, and we are mentally more exhausted because we didn’t really do the heavy lifting, meaning we don’t really understand what was generated. And potentially also didn’t learn anything new.

With that in mind, we need to be careful when to use the new tools, certainly not always, but there are many ways. So how else should we use agents and AI as data engineers and knowledge workers?

[!note] Plan Mode Support

Besides Claude, Plan Mode is widely adopted across AI coding assistants including Cursor (October 2025), Windsurf (June 2025), GitHub Copilot (VS Code, Visual Studio, JetBrains, Eclipse, Xcode), Lovable, Bolt.new, and Replit (September 2025). Everyone is following a similar pattern of letting AI analyze, ask clarifying questions, and propose structured implementation plans before writing any code.

Plan Mode: And How We Work Best with AI Agents

This is how we humans work best as well. We make a plan, and then execute it and adjust along the way. But it’s also a great way to work with juniors, and in that sense, AI agents.

Because we say what we want in an abstract manner, the agent says what it would do in a plan form (just a markdown file, markdown runs the world these days), and then we as the senior, or the designer or architect can see if it missed our interpretation (as language is not precise), and we work on a great plan with all the details. This way we know it does what we expect it to do. And then it goes off and does it autonomously with access to the terminal and all command line tools.

But there’s one more factor, it’s the human factor. Whatever it builds, it builds on trained data. So it will use what most people use. Which might be ok for most cases, but maybe not if you want to build something unique, innovative. That’s why I think for most writers, it’s not the right tool to let it write the stuff for us. Just for that fact, but even more so, the character and soul of the person gets stripped away. The quirky things someone does, which make them who they are, that takes away from the fun of writing.

Obviously in coding, this is not the same. Except if you are another programmer and need to read the code, no? Because any data engineer would love to read the code from a human rather than an AI, it’s kind of boring. But maybe it just needs to do the job, and not all human code is beautiful too, right?

[!note] See the Prompt for the Web App

If you want to know how I built the web app without having experience in Next.js, I am sharing the initial prompt with plan mode that could be interesting. The summary of the full session (ca. 3-4 hours) is at build-summary.md.

Where Are We Heading?

So what about data engineering? Where are we today?

As I have written extensively about at Self-serve BI thanks to AI or using it for data modeling along with semantics, speed, and stewardship, humans still need to be in the loop, and we need to be careful to not generate too much (ingestion logic, business logic, general code, or dashboards) that is unmaintainable or never needed in the first place.

On the other hand, there’s no definitive answer right now, we are all just figuring it out. That’s why some say it’s the most exciting times, because everything is supposedly going to change. Andrej Karpathy said:

Clearly some powerful alien tool was handed around except it comes with no manual and everyone has to figure out how to hold it and operate it, while the resulting magnitude 9 earthquake is rocking the profession.

As a writer but also data engineer, I find it most useful when it suggests notes and ideas I have forgotten about that are relevant to my current task at hand. Or a snippet of code.

Repeating Code Snippets over and over

How many times have we written an ingestion pipeline that does the same thing just for a different source? Written an incremental update pipeline, or a full load, or implemented Slowly Changing Dimensions (Type 2).

Wouldn’t it be great to have a tool that helps us remember and suggest code that worked for a problem at hand? No wonder Windows has a built-in Windows Recall feature that takes snapshots of everything we do, so we can see and remember what we did. Google traces where we went on Google Maps Timeline, and so on. Not saying all of these are good, but clearly there’s a need for it.

Vibe Coding

Mostly these tasks are called vibe coding these days. I believe that vibe coding is best when you have an existing framework present and it can extend it. E.g. your website skeleton that already has a pre-existing structure is much better than starting from scratch, especially maintainability-wise.

Also, the more it has to predict in the future, the more likely it will introduce errors, compared to you providing a big skeleton with all the needed files and just extending on functionality.

This is the same for data engineering too. Declarative Data Stack, YAML Engineer is exactly that. A well-designed YAML that has a powerful system in the backend can go a long way with an agentic and vibe-coded approach.

It’s similar to Spec Driven Development (SDD), which is when we write our instructions in claude.md and Claude or any AI agents implement this. Also what Esco Obong said about what they do at Airbnb: the hard part is coming up with the spec, talking to business, etc. The coding part is the small part.

And this is also where the human is still dearly needed in my opinion. Human in the seat and config-driven development is what it comes down to with AI agents. Plus, AI models have a context limit. Sure, we humans do too, but we can think more across domains and understand intuitive things that might not work for a statistical model.

This shows how that works, and why Markdown is in the middle of everything. Not only for the knowledge, but also to build and develop things.

[!tip] Lifehack for Prompting

Always keep it simple, because it’s easy to make it complex. The true beauty lies in making it simple, which is something agents are not good at.

Use MCP

A key was using MotherDuck MCP with a direct connection from Claude Code to the database while prompting the initial version. Claude could directly query the database and its columns to implement the actual web app (see the initial prompt here).

Meaning Claude (in my case) could just query the database, use SHOW TABLES, select them, and extract their data types. And more, learning about the content and graph relationships that I had built in the first part.

So Claude could easily build a first version based on my instructions and existing DuckDB database. I also shared the great docs to build Customer-Facing Analytics Guide in a (3-tier Architecture).

With that, I almost had my web app ready with a single plan mode prompt.

[!example] Claude supports LSP now

As code editors do, Claude also supports LSP (Language Server Protocol). This helps speed up Claude to read the code more efficiently, doing lookups by jumping to references or definitions instead of searching its way through the code. It might also understand the code better as it has a language server to use.

Conclusion

Building this tool reminded me again how powerful DuckDB and MotherDuck are. It’s a Swiss Army knife database that can handle unique tasks and simplify my note-taking by providing a serverless database for querying my embeddings.

Now I have a powerful tool to search for related notes when I need to solve a problem, or to find relevant notes in my own second brain. The hidden connections this tool surfaces are valuable only because they’re my connections, my thinking, not just crawled information on the internet. And not only that, I can even provide a minimal but useful web app for you to search my public notes, too.

As for the AI agents that helped build it: they got me there faster, but only because I stayed in the loop. Let them run without direction, and you’ll get a thousand lines solving the wrong problem. To me, the “human” architect is still needed.

Other implementations I have collected over the years or came across while building this that might be helpful if you want to build something similar.

If you have many more files and embeddings that need to be created, follow the Using DuckDB for Embeddings and Vector Search article that runs on the GPU, creating embeddings for 2.85M Wikipedia articles. He used the Arrow/GPU acceleration and batch inserts via Arrow.

Some more links and repos I found interesting:

• Scalable Embeddings & Vector Search
  • Using DuckDB for Embeddings and Vector Search: Tutorial on GPU-accelerated vector search that created embeddings for 2.85M Wikipedia articles using Arrow batch inserts and HNSW indexing.
• Local-First Search Tools for Markdown
  • qmd: Tobias Lütke’s CLI search engine combining BM25, vector search, and LLM re-ranking—all local via Ollama, works with plain markdown (no wikilinks needed).
• Obsidian AI Assistants
  • Obsidian Copilot: A popular Obsidian AI plugin (6.1k+ stars) with vault chat, agent mode, and image/PDF/web processing—no index required for basic search.
  • Chat with Your ENTIRE Obsidian Vault OFFLINE (YouTube): Video walkthrough of offline Obsidian vault chat with Claude 3 integration.
• RAG Frameworks & Libraries
  • Quivr: YC-backed opinionated RAG framework (38.6k+ stars) supporting any LLM, any vectorstore, and any file type with YAML-configured workflows.
  • LennyHub RAG: Complete RAG implementation on 297 podcast transcripts with knowledge graph extraction, Qdrant storage, and interactive network visualization.
• AI-Assisted Development in Production
  • Esco Obong on AI Coding at Airbnb (LinkedIn): Airbnb engineer shares writing 99% of code with LLMs, noting that code is “only a small part of the actual work.”
• My List of Obsidian Related RAGs: Second Brain Assistant with Obsidian

Full article published at MotherDuck.com - written as part of my services

Back then, I had a checklist of basics I needed before I could commit to Linux as a daily driver: Obsidian, a Raycast-like launcher for fuzzy finding files and folders, screenshots (Snagit), daylight adjustment (f.lux), calendar events in the top bar. Those were quick wins.

Eight months later, I’ve gone through many more challenges and learnings. In this post, I’ll share which apps replaced my heavily integrated macOS workflow, what my Omarchy workflow looks like now, and — honestly — what still doesn’t quite work.

Apps that Replaced My macOS Apps on Linux

Let’s start with which apps and how I changed some of my workflow now in Linux.

Below list goes from complex Raycast replacement that was integrated into my whole workflow with search through files, calculator, emojis to calendar, daylight gamma correction for night sessions to PDF viewer that replaces Finder to sharing screen with Linux window picker, and much more.

It continues with running Windows on Linux with a simple install toggle and finding the right hardware, before I create a conclusion of these initial months using Linux full time for my business and also privately.

App Launcher and Raycast Replacement: Fuzzy Search, File Search, Clipboard, Math, and so on

One of the first apps to replace that most have, and that I also used, is Raycast. It’s an app I couldn’t live without, not only for the fuzzy finder but also for quick calculations, searching files, and clipboard manager.

With [[Walker Launcher]] I found the perfect replacement which has this all included and works like a charm.

Search file content with spotlight - Find files with Walker with built-in preview: ![[img_Switched from macOS to Linux- 6 months in_1770740249453.webp]]

Opening its containing folder or file with the default program. This is how I search and find anything compared to manually browsing through file explorer. Find any files within seconds with built-in search of Walker (Before I found Walker, I built my own one).

Emojis quick search. It comes with Walker built-in too, but I have my own script so I can find emojis faster as I can change the search term. Very simple, but powerful
![[img_Switched from macOS to Linux- 6 months in_1770741956438.webp]]

Clipboard managers, of which there are several, but Walker comes with one built-in too. Including search and image preview:

Other dedicated clipboard managers are cliphist or Clipse. There are also other Raycast-compatible launchers for Linux such as flare, Rofi, and many more.

Keyboard Shortcuts and Quick Symbols

I used Kanata for integration of advanced features to switch between my keyboards and some of the advanced use cases such as using CAPS LOCK for vim-like movements. I use caps + hjkl to move left, down, up and right with the respective arrow keys as almost all programs work with arrow keys. Also F1-F12 functions with caps+1 for F1.

For simple replacements, I used XCompose to write Umlauts (äöü and special symbols —«» and more). I used Karabiner-Elements heavily, and Kanata solved it for me, see my configs at dotfiles.ssp.sh/kanata.

Backups and Data Sync

Time machine on macOS was great. I used sync.com for dropbox-like sync on macOS too. Neither worked on Linux. So I switched to Filen, which has a similar setup and stores the data encrypted, and hosted in Germany. I’m using Stow for all my dotfiles stored in Git. It’s great, check them at dotfiles.ssp.sh.

I back up my images, personal documents, or scripts also with rsync-scripts to save to my homeserver and encrypted drive on Vultr. See more Tech Independence.

I also looked at NextCloud for hosting it myself, but for now I just need something that works. As Filen is an Electron app, it just works everywhere.

Calendar

Calendar is one thing everyone uses, and I used Cron Calendar (later acquired by Notion) a lot, and wanted a good replacement for Linux. Though I use calendar.google.com often on the web.

But the best replacement I found was Morgen (built in Switzerland) and is made for Linux first. It has a great preview inside the top bar too and timezones built-in.

![[img_Switched from macOS to Linux- 6 months in_1770740582450.webp]]

Time zones can also be activated by hovering on the time on the left:
![[img_Switched from macOS to Linux- 6 months in_1770740650778.webp]]

Daylight and Gamma light Adjustment

Sunlight adjustment like f:lux. Omarchy comes with one included right now, but I also used wlsunset with wlsunset -l 47.4095 -L 8.5514 -t 3500 -T 6500, that does the job well.

Hibernation and Suspending Computer

Hibernation and suspending is something that you take for granted on other operating systems. But on Linux it’s trickier, so it didn’t work out of the box. In the meantime, it comes built into Omarchy.

Presenting with External Projectors and Screens

Presentations and recognition of presenters and screens. I only had one presentation, but tried many external monitors, and Hyprland (which is responsible for recognizing screens hyprctl monitors) works just like macOS by auto-recognizing them.

Even better, I have shortcuts to make them automatically align at the right position. Or use hyprmon (one of the great [[TUIs]]) when I need to do it manually.

PDF Merger

PDF Arranger for merging multiple PDFs into one or rotating pages of a PDF. It’s open source and better than macOS Preview.

Need Anything More? Just Build (vibe code) it Yourself

If you need something, you just build it with [[Claude Code]] and integrate it into your laptop.

No need to ask Mr. Bill Gates or Tim Cook to integrate it. For example, I needed an edge light for video calls, or saw someone who had this. I liked it, not that I really needed it (but one day it might be helpful 😀). I’ve built a small custom tool that works out of the box with Hyprland for my future video calls.

Screen Sharing Works Differently

For example, screen sharing is not as straightforward because you get a very old frame to pick your windows or output or regions. On top of that, you usually need to pick twice and only the second pick will count. This was very confusing and I documented a fix and how it looks at [[Screen Sharing on Wayland (hyprland) with Chrome]].

But with the latest updates of Omarchy, that has also been solved and it works out of the box and looks beautiful now:

Others: Virtual Envs, Remote Desktop and Adding Printers

For virtual environments, I’m using Mise, as it comes pre-installed on Omarchy. Before, I used asdf.

Remote desktop to virtual desktops works great with xfreerdp3, which connects well to the Windows VM.

Need to import images from camera? Not as UI-driven as on Mac or Windows, but amazingly simple and fast with gphoto, see [[Import Files on Arch Linux (gphoto)]].

Adding printers might be needed at some point. This can be done UI-driven with system-config-printer - CUPS configuration tool. Or do it the terminal way with lpadmin, see [[Adding Printer on Linux]] for more information.

Running Microsoft Windows inside Linux

A big one is to run another operating system, in this case Windows, as part of your OS. E.g. I use Microsoft Office often, so I can quickly start up Windows with Office when needed.

The best part is that it uses only Docker, meaning easy setup, separated from my configs. It’s a single-click setup taking 15 seconds.

Installing and integrating seamlessly in a Docker VM works superbly with Omarchy. I submitted a PR to Omarchy to make this available to everyone. The built-in version in Omarchy (the first version) was done by me, and it was merged into core and is now available to everyone.

It’s now easier to run Windows on Linux than natively on a Windows machine 😉.

Finding the Right Hardware: The Reasons why not to Switch

Everyone knows the stereotypes about Linux. WiFi won’t work, Bluetooth won’t connect, constant interruptions. And beyond that, there’s the hardware fear, that you simply can’t match what Apple offers. A common sentiment:

I’m currently with this dilemma. I’m an experienced Linux user, but over the years gravitated towards Macs (especially M-series) and unfortunately they do make better hardware, at least for my use 🙈 I just can’t move to a machine with a much worse battery life, display, webcam, speakers etc. I know some good Linux-friendly laptops exist, but it’s still a downgrade, for me. If someone made better hardware, I’d probably jump over right away. Tweet

I thought the same. The great keyboard, camera, speakers, trackpad, battery. Apple just nails the whole package. But what I found is that I didn’t actually have to downgrade.

I started with a Lenovo ThinkBook 14 G7 ARP (AMD) with 32 GB RAM. Great build quality, beautiful look, and the keyboard surprised me, with much more travel and grip than the MacBook. See more on Part 1.

Once I realized this would become my daily driver, I searched for something more powerful for data engineering work and landed on a Tuxedo InfinityBook Pro 14 Gen10 AMD with 128 GB (!!) RAM, an AMD Ryzen AI 9 HX 370, and AMD Radeon 890M.

First impressions: super smooth, even snappier than the Lenovo. Obsidian and other apps feel a tiny bit faster. The 3K 500-nit display is stunning. Crisp, bright, better than my external 4K monitor. And it’s matte, which I’d forgotten I actually prefer. It works outside, no glare. The Lenovo’s anti-glare screen was equally great in that regard.

The keyboard has less travel than the Lenovo ThinkBook (which I really loved), more like a MacBook, which is fine but feels slightly cheap. I mostly use external keyboards anyway. The fingerprint reader is also missing, which I’d grown to love on both MacBooks and the Lenovo, where it worked flawlessly on Linux. The trackpad is smooth and great to work with daily, though palm detection caused some cursor jumping in the first days, not as good as Apple’s, but perfectly usable.

Battery life was a pleasant surprise. My Tuxedo (80Wh) delivered battery life comparable to my M1 Max MacBook. I spent a whole afternoon in the library and it was still above 70%.

With Omarchy, everything just worked out of the box. No WiFi or Bluetooth issues, speakers with sound, all good.

But it’s not perfect, by far. I get some strange popping sounds from the laptop, mostly after hibernating once or twice. Not sure why, and it probably shouldn’t happen. There are many other [[Notebook & Desktops for Linux]] to choose from, Framework being one, but choosing is still tricky, as chipset and GPU support on Linux matters, and you want something state-of-the-art.

Another side effect, as Kevin says of not having an expensive Macbook:

My favorite part aside from customization is just that i don’t care about my machine at all: it gets lost? breaks? stolen? i get a new machine, run 1 command and everything is back exactly as i left it. Macs on the other hand are expensive to buy and repair which makes people worry and worry = less peace of mind.

[!example] Follow the evolution on Social Media

The whole story I documented in threads on Twitter and on Bluesky, follow these to see the history and events in they happened.

Conclusion of Using Linux for 8+ Months

After using Windows since 2003 and macOS for more than 15 years, how do I feel after 8 months on Linux?

Things mostly work great, but need a little tinkering to begin with, or work differently. The biggest difference, which I like a lot, is a more terminal-native workflow. Closer to the command line. Using lots of [[TUIs]].

When I started, I just wanted the same as I had on macOS. After getting familiar with the new environment, with all the small utilities, tools and programs Linux has, I got many more tools to choose from. Sometimes much better, though terminal-based, but fast and direct. Sometimes you obviously miss a tool that has no replacement (for me still Snagit).

Besides the obvious (terminal-native, best-in-class Tiling Window Manager with Hyprland, no-latency navigation), there’s something harder to put into words. The OS is what we use every day, so when you can quickly fix or change a small thing to give you more joy or more productivity, it might just put a smile on your face whenever you use that feature. At least it still does for me. And since all my configs live in dotfiles and my data syncs externally via Filen and Obsidian, setting up a new machine is a single command.

What I Thought I’d Miss vs. What I Actually Miss

Before I switched, I thought I’d miss all my Setapp, my MacBook hardware, and the stability of just working. Most of my apps work on my new machine and even better software-wise, so I’m still quite happy to have made the switch, even more so watching macOS get slower with each install without real benefit (looking at the Liquid Glass update) and Windows stuffing Copilot into Notepad and recording your screen with Recall.

What I actually miss are simpler things: the stability of having calls everywhere all the time with Apple reliability and inbuilt mic/speaker/camera. A crash because the GPU is fails¹ after hibernation right before an important meeting. Hibernation and suspending was quite a battle to get working, but it seems to just work now.

What I like about Linux: it might not work out of the box for every laptop or every program, but you can actually fix it, and from that moment you know the problem, you learned something about computers, and the error will not appear again. Unlike other operating systems that change stuff you set in settings for a reason, only to learn that certain updates turned that checkmark back on.

Tinkering and Troubleshooting: Not Everything Just Works

Probably, without [[Claude Code]], I wouldn’t have made the switch, or I would have made it, but probably wouldn’t have stayed. When something happens, e.g., a crash out of nowhere, I just open Claude and say: I had a crash, I am running Arch Linux, please check the logs what went wrong. And what I get is a full analysis of what went wrong, some fixes and suggestions. Knowing that Linux has 100 different setups, different drivers for every hardware, this is a non-negotiable lifesaver.

With Linux you also have to troubleshoot bugs, but at least it’s free software and open source, and honestly, they seem even less frequent than with commercial, paid products these days.

If you’ve read this far, thank you. What’s your experience, are you thinking about switching, or already on Linux? Let me know anywhere on Bluesky, or Twitter.

Again, if you want to watch a full video workflow, check my short video about it: Omarchy Arch Tiling Window Workflow (macOS comparison) - YouTube. Or my macOS workflow as a comparison and what I switched from.

Appendix: Troubleshooting and Things That Didn’t Work So Well, or I Had Already Fixed

GPU Crashes (AMD Radeon 890M). This was the biggest recurring issue. The GPU’s MES (Micro Engine Scheduler) would become unresponsive and crash the entire system, triggered by Brave browser, Google Meet video calls, and Kdenlive video encoding. The root cause is that the Radeon 890M (gfx1150/RDNA 3.5) is still very new, and driver support on bleeding-edge kernels (6.17–6.18) is immature. Solutions included disabling hardware acceleration in Brave (brave://settings/system), adding kernel parameters (amdgpu.gpu_recovery=1 amdgpu.noretry=0 amdgpu.ip_block_mask=0xfffff7ff), and considering the LTS kernel as fallback. The community is tracking this on Framework forums and AMD’s GitLab.

Keyboard Freezing After Suspend (Tuxedo). The internal keyboard would stop working after suspend/resume cycles due to a firmware bug in the keyboard controller (i8042). Fixed by adding i8042.nomux=1 i8042.reset=1 i8042.noloop=1 i8042.nopnp=1 to the kernel command line in /etc/default/limine and regenerating the UKI with sudo mkinitcpio -P. Shared the solution on Reddit.

Hibernation Not Resuming. After suspend-then-hibernate (triggered by closing the lid), the laptop wouldn’t resume and required a fresh boot. The cause was missing resume= and resume_offset= kernel parameters. Omarchy’s hibernation setup script added the mkinitcpio hook but never added the actual kernel parameters. Fixed by calculating the swap offset (sudo btrfs inspect-internal map-swapfile -r /swap/swapfile) and adding resume=/dev/mapper/root resume_offset=<offset> to /etc/default/limine. Documented the fix in this Omarchy issue.

Thermal Throttling (Lenovo). The Lenovo ThinkBook would hit 99°C and become unusable during video calls. Turned out the bottom intake vents were blocked when the laptop sat flat on a desk. Simply elevating the laptop dropped temps to 73–77°C and performance was completely fine, even running stress tests while screen sharing. A laptop stand solved it permanently.

WiFi Speed Drops (Tuxedo, Intel AX210). Speeds dropped to 2–72 Mbps after a system update. Root cause was a bug in linux-firmware-intel version 20251125 that caused the Intel AX210 card to negotiate very low RX bitrates. Fixed by downgrading to the October firmware (sudo pacman -U /var/cache/pacman/pkg/linux-firmware-intel-20251021-1-any.pkg.tar.zst), disabling WiFi power save permanently via a systemd service, and adding IgnorePkg = linux-firmware-intel to /etc/pacman.conf until a proper fix ships.

Keyring/Brave Re-login on Every Boot. Brave asked for login credentials after every reboot because the gnome-keyring file kept getting corrupted. This was caused by SDDM autologin. Without entering a password at login, PAM can’t unlock the keyring. The ultimate fix was launching Brave with --password-store=basic in the autostart config. Documented in this Omarchy discussion.

Sudoers Misconfiguration. While adding a NOPASSWD rule for a keyboard-switching script, I accidentally broke sudo access entirely. Had to boot into recovery/single-user mode to fix /etc/sudoers. Lesson learned: always have sspaeti ALL=(ALL) ALL as a separate line and be very careful with visudo. Documented the recovery process in an emergency recovery guide.

Screen Recording VFR Issues. Omarchy’s screen recorder (gpu-screen-recorder) produces variable frame rate videos by default, which Kdenlive can’t edit properly. The fix is adding -fm cfr to the recording command. Additionally, Kdenlive’s VAAPI hardware transcoding crashed the GPU (same MES issue), so software encoding (libx264) is needed for now.

CPU Fan Noise When Plugged In. The system switched to “performance” CPU governor when plugged in, causing constant full-speed fans even at low load. Fixed via powerprofilesctl set balanced or through Omarchy’s built-in power settings menu.

Two questions interested me most. Why do joins seem to be faster with StarRocks than with other OLAP databases like ClickHouse, Druid, or Pinot? And how can it deliver fast responses even when data sits on cold storage like S3?

This article covers their answers, the architectural innovations behind StarRocks (colocated joins, caching, cost-based optimizer), and the tradeoffs you should know before evaluating it for your own analytics needs.

Introduction

The modern analytics challenge is to serve queries fast while having data on a data lake on S3, CloudFlare R2 or other places. Usually, the advantages of data lakes (storing data easily without much care about synchronizing database schemas among tables or validating bad rows) result in the opposite of fast queries. This is acceptable if the data is for internal teams or non-critical data. But customers, or also business and domain experts, usually don’t want to wait minutes for a database query result, only to then need to add another filter or column and wait another few minutes.

This workflow is even less efficient with AI agents when they initially autonomously query and narrow down the domain for you. If these queries are slow, the interaction with an agent or chatbot will be even slower.

Enter StarRocks, capable of querying lakehouses - Iceberg, Delta Lake, Hudi, and Hive - in place, without moving data. That’s the pitch. Let’s see how that works.

What is StarRocks?

But first, what is StarRocks? The definition and short version from their docs:

StarRocks is a next-gen, high-performance analytical data warehouse that enables real-time, multi-dimensional, and highly concurrent data analysis.

StarRocks has an MPP architecture and is equipped with a fully vectorized execution engine, a columnar storage engine that supports real-time updates, and is powered by a rich set of features including a fully-customized cost-based optimizer (CBO), intelligent materialized view and more.

StarRocks supports real-time and batch data ingestion from a variety of data sources. It also allows you to directly analyze data stored in data lakes with zero data migration.

StarRocks started as a fork of Apache Doris, but claims to have rewritten 90% of its code since then, mainly to improve performance, stability, usability, etc.

In the past three years, the StarRocks team has replaced the query optimizer with a brand new Cost Based Optimizer to eliminate de-normalization, implemented a Vectorized Query Engine to improve query performance, designed a Primary Key Data Model to better handle real-time analytics scenarios, released Intelligent Materialized Views to simplify data pipelines, and rolled out many other breakthroughs.

Is it a Real-Time Analytics Database, OLAP, Data Warehouse?

StarRocks is also an OLAP system with fast sub-second response times, focusing solely on analytics use cases, not transactional processing. But StarRocks is more than that, as it supports joins, so the terms lakehouse architecture and data warehouse come into play, potentially suiting BI use cases better.

StarRocks implements and provides the capabilities of the MySQL protocol, and with its intelligent materialized views and a newly designed cost-based optimizer (CBO) built-in, it’s a powerful tool that should save you a lot of engineering time.

StarRocks promises one system that can power multiple analytical scenarios, reducing system complexity with Frontend Engine (FE) and Backend Engine (BE) nodes.

Who is Using StarRocks?

There are some big names, some of which we will interview after this chapter. The biggest are Pinterest, Coinbase, Naver, Fresha, Lenovo, Expedia, Trip.com and many more.

Why StarRocks?

So besides other real-time and OLAP databases such as ClickHouse, Pinot, and Druid, when do you choose StarRocks? What are the use cases?

First, let’s see what real StarRocks users that I have interviewed say, before we analyze the technology decisions and techniques that make StarRocks a valid option.

A Simplified Cluster Topology with Support for Fast, Distributed Joins

Key selling points are a simpler architecture, native fast distributed joins, intelligent Materialized Views that can refresh complex joins, and federation as a fast compute engine on top of data lakes and lakehouses with data on object storage.

We’ll hear from Coinbase, Pinterest, and Fresha themselves about why they use it. And we’ll go into joins, colocation, and caching mechanisms later.

But before that, let’s understand the general architecture with two node types, Backend Engines (BE) and Frontend Engines (FE), and why it’s perceived as simpler than others. The backend nodes can be both BEs and CNs (Compute Nodes). The backend nodes support two storage variants: one with local storage in the BE nodes and one with external storage such as S3/HDFS.

In the shared-nothing architecture, each BE stores a portion of the data on its local storage, and with shared-data, all data is on object storage or HDFS and each CN has only cache on local storage. The default is shared-nothing, meaning direct access to local data on the BE node, but the future is Shared-Data mode to be more cloud-native and reading directly off object store. The convenient thing is that you choose where the data is stored based on your needs.

Keep this in mind while we go through more of the interviews and pros and cons. There are more details in the “StarRocks Technology Decisions” chapter, but let’s first dive into the valuable insights from actual users and companies who explicitly chose this architecture, and why they did so.

Why Coinbase Chose StarRocks

In this part, Eric Sun, Head of Data Platform + Datastores at Coinbase, long-time LinkedIn Manager and experienced with data systems, gave me the pleasure of interviewing him and learning from his expertise. Below are his answers to questions regarding StarRocks at Coinbase, and anecdotes from his personal past.

Coinbase is the largest cryptocurrency exchange in America with over 100 million users and 200 cryptocurrencies. This means they process billions of transactions daily, even more so in short specific periods when the market is active and everyone wants to sell or buy.

The Origin Story for StarRocks at Coinbase

That’s why they started using StarRocks. Originally, 90% of the workloads were on Snowflake, but they needed a faster Operational Data Store (ODS) engine for crypto data services.

Other solutions such as TiDB were tested, but it was too disruptive to bring another transaction processing database into the company. Compared to Pinot, ClickHouse, and Druid, they ultimately chose StarRocks because of these strengths:

• Ingest with light transformation ⇨ pre-aggregate ⇨ analytics lifecycle is still too long/slow for other DWHs like Snowflake and Databricks
• Query performance for ad-hoc and online services requires pre-aggregated, pre-warmed, and pre-cached engines like StarRocks, Doris, Trino, or ClickHouse. Basically balancing both fast data ingestion and join capability, but also near real-time data serving.
• Costs of Snowflake and Databricks are both too high for the use case

And they quickly expanded StarRocks to use for trade/exchange data, event/clickstream, and as a Facebook Scuba alternative.

The JOIN Question

A key and distinctive advantage that StarRocks seems to have over its competitors is the ability to JOIN. Almost like a data warehouse, but with the speed of an OLAP database. I asked Eric how they use joins and if the feature lives up to its promises.

The question that interested me most was: “Can you join and perform simple ETL without persisting data in an intermediate step (like a data mart)? Is that all done automatically with StarRocks? And if so, how can this be fast enough?”

Because you can’t overcome the laws of physics, reading from S3 is just slow. He said the two key techniques are hash-distributed partition + colocation for multiple big tables to join with minimal overhead. He says “this is nothing new, but most other engines, including Snowflake, Databricks, and ClickHouse, have not incorporated these two simple-yet-effective traits”.

He continued with “S3 is slow, so frequently queried data chunks must be automatically cached to the BE (backend) nodes of StarRocks (after the warm-up queries) in the ‘shared-data’ mode”.

Further, I asked him: “But how do you model the data flow then? How does StarRocks fit the picture, are you doing any ETL before landing in StarRocks?”

He surprised me with data modeling still being the key:

Transactional Processing (OLTP or relational) data models in most cases are not a good fit for StarRocks or ClickHouse. But if the join/foreign keys are clearly defined in the data models, StarRocks can leverage Colocate Join to reduce pre-join via ETL.

However, pre-join can always bring visible benefits to query performance and data quality. The point here is about how much percentage data models can be efficiently served via StarRocks without streaming joins in Flink/Spark which requires highly-skilled engineers.

He continues with denormalization and colocation by dimension:

Event data are typically denormalized to some extent, so we typically just need to colocate the big event table with the big User/Customer/Product dimension table via the USER_ID / CUSTOMER_ID / PRODUCT_ID

We will explore colocate joins in a bit, as this seems to be a key part of the speed for joins. Also, the distinction between when to stream and when to use StarRocks is super helpful and shows again that almost all good designs come down to good data architecture and modeling your data flow.

On top of that, Eric mentioned that “query planning and data modeling is the key to the success of their project”. For example, they use Kafka as a Sink and land data before going into StarRocks as one modeling choice for certain data.

Colocated JOINS: How They Work

Interestingly, colocating dimensions with the large fact tables is a noteworthy approach. Let’s quickly explore how these colocated joins work in detail to understand the importance of this, before we get back to the interview.

Colocate Join lets equi-joins run locally by ensuring tables share the same bucketing key, bucket count, and replica placement so corresponding bucket copies reside on the same backend nodes, avoiding network shuffle or broadcast overhead.

Tables that should join together are organized into a Colocation Group (CG) with the same schema (Colocation Group Schema (CGS)), consisting of these three properties that must be the same for all tables:

• Same bucketing key (type and order). e.g., both tables use customer_id INT
• Same number of buckets. e.g., both have 8 buckets
• Same replica count. e.g., both have 3 replicas

This guarantees that rows with the same customer_id always land on the same node, making joins local.

There’s much more. Check out the docs on Colocated joins and Schema Tuning Recipes. But this also shows that there’s a lot of planning involved in how you model your data, as these settings need to be consistent across tables and optimized for your use case and data.

Trade-offs: Key for Partitions

There are always tradeoffs. As seen with the colocation and schema groups, you need to define what to colocate on. You need to know your data well.

It’s the same as we always did in our work, deciding what to partition on, but here it’s done more holistically across nodes and tables. It’s a tradeoff and a choice which columns you do exactly that for.

For example, Coinbase with its Bitcoin data has distinct addresses and dates, so you could either partition by range on such high-cardinality addresses or by blockchain timestamp, but you can’t use both. You need to decide.

On distribution key tradeoffs, Eric says:

You can only optimize for one of them and leverage the index for the other one. That’s a trade-off you have to do.

This also fits into how you model the data so you don’t need to join, or when you do, you can join efficiently. StarRocks excels at joins using primary keys or unique keys.

For Coinbase’s blockchain use case with high-cardinality address columns, Eric and Xinyu recommend:

1. Partition by timestamp (monthly) rather than by address, as high cardinality makes address-based partitioning operationally unscalable.
2. Distribute by only one address column (from OR to, not both). If you distribute by both, queries must use AND predicates, which rarely match actual query patterns.
3. Use ORDER BY on the distribution column to avoid needing bitmap indexes on that column.
4. Create a secondary lean table optimized for the other address column, then join back using compound keys (transaction ID + timestamp + block hash).

Struggles with StarRocks

When asked what they lost or struggled with when switching to StarRocks, Eric said that the community is much smaller than ClickHouse’s, for example. A well-known community and perception in the US is important, as most people in the Bay Area and Seattle area have still never heard about StarRocks.

Compared to ClickHouse, which is buying analytical solutions and trying to be a cohesive software stack, StarRocks seems to take a simpler and slower approach, he comments.

He also missed the stability compared to Snowflake, as new releases sometimes introduce dozens of small errors in certain areas such as deployment, system metadata, and elastic compute. But overall, it is mainly popularity and acceptance from the user community, because “people are more willing to try and learn the technology that they have heard about or their friends can mention”, he says.

Forward-Looking: Lakehouse Architectures and Benchmarks

When asked about using StarRocks with Iceberg/Lakehouse solutions and loading data directly, Eric said: “Both: hot and recent data (2 weeks ~ 3 months) are stored in StarRocks native layout on S3, such as partition, index, bucketing, colocation, …, while the cold historical data are federated from Iceberg/Delta to share the same data partitions from Lakehouse. Iceberg can’t deliver the performance compared with the native storage format”.

In terms of comparison and benchmarks, Eric says that StarRocks significantly outperformed ClickHouse in their TPC-H 1TB benchmarks. ClickHouse failed 12 of 22 queries due to out-of-memory errors, particularly on join-heavy queries. The data was 10 blockchains with 300+ tables and 573 billion rows.

But the competition is still ongoing, he says: “Simply put, StarRocks naturally fits much better for multi-table join scenarios especially for e-commerce and finance sectors. And ClickHouse has better out-of-box templates for observability use cases”.

He continues that ClickHouse is harder to maintain, as there are so many knobs and tweaks, whereas StarRocks on average is much simpler for the engineering team to understand and learn, and more manageable.

When asked what they can learn from each other, Eric said this:

• StarRocks should learn from ClickHouse: Memory Table, Integration / Connector with other partners, and rich complex/advanced UDF/UDAF
• ClickHouse should learn from StarRocks: Sophisticated cost-based optimizer (CBO), multi-table Materialized View, primary key table optimization for DML, Concurrent Queries and Join Join Join 🙂

Why Fresha Switched to StarRocks?

The second interview with Anton Borisov, an experienced Principal Data Architect at Fresha and a heavy user of StarRocks who is building their own tooling on top of it, gives us more valuable insights. Anton has a strong background with relational databases, specifically Postgres, and has worked with distributed OLAP systems. Fresha is the world’s leading marketplace platform for the beauty, wellness, and self-care industry, trusted by millions of consumers and businesses worldwide.

First, we talked about why they switched, and it was the same pattern as with Coinbase. Customer-facing analytics built on top of dbt materialization and batched into Snowflake got too slow (every 20 minutes).

Anton was a big ClickHouse user and wanted to use it first. But then they discovered StarRocks and found that the joins would simplify their architecture. Especially with dbt where they ended up with lots of different layers, CTEs, joins, etc. So with StarRocks, they threw the same SQL at it and got a sense of how it performed and how feasible it was. That was their first baseline of what’s possible and how fast. They achieved that rather quickly and could optimize on top of it. Whereas with ClickHouse it was much harder to set up a first working baseline to then iterate and improve on.

As the first baseline was 4 seconds with lots of joins, it was a great start from which they could tweak colocation and optimize data flow.

Anton told me that Fresha uses two pipelines or ways of ingestion into StarRocks:

1. Streaming data that is slightly pre-aggregated with Flink and then ingested directly into StarRocks. StarRocks then handles the optimal storage with CN (Compute Nodes), which are stateless and fetch data from S3 in shared-data mode. Then you have cache in memory and disk. You could also use Kafka Connect, Debezium, and many other ways to ingest.
2. Data from S3. Here the workflow is Snowflake -> Spark joins and export to Iceberg -> StarRocks
   • Important to note: Spark gave them the best option to add as much metadata as possible, like Apache Puffins and general Iceberg metadata, for StarRocks to quickly read from, even if not in hot storage.
   • This approach is done on a batch schedule, usually once a day. Also interestingly, they might duplicate certain datasets and store them in different sorting orders (e.g., one on dimension customer, one on dimension region), just to have StarRocks optimize reading on cold and cheaper S3 storage for different dashboards. You pay extra for S3, but you can avoid expensive hot cache if you want. You can still let StarRocks load some of it into local cache to speed things up, say if there’s an event coming up.
   • This architecture makes it flexible without having to change data pipelines.
   • Fresha treats Iceberg tables as mostly immutable: data is ingested append-only, and they don’t continuously apply row-level corrections. Instead, they accumulate corrections and periodically re-ingest/merge them back into Iceberg for the workloads that need corrected history. The target model is that older (non-operational) data remains stable, with only controlled correction cycles rather than ongoing mutations.

The way it works is that data from Flink lands in RAM first, then gets offloaded into cache. There is an option to specify the cache-to-memory ratio, but you can’t control it directly, so you need to tweak the workload. They use the shared-data mode for this. Remember the above image about the architecture overview.

Regarding joins, Anton said that they just work as you’d expect from a data warehouse. The speed heavily depends on how much “money” you invest, meaning how many CN nodes you scale up when reading from S3, for example. And because CN nodes in the shared-data architecture are stateless, it’s easy to scale up more nodes. A great detail Anton shared: these CN nodes do RPC (bRPC to be specific) calls with each other to communicate.

You can also use Materialized Views for Joins and Ingestions, but they do most in Flink or Spark.

How to Handle Data Updates

Data is updated with a continuous streaming job like Flink. StarRocks has a compaction job that optimizes the data that has been updated. It’s an internal process where you configure some limits and times for it. Colocation, though, is handled a bit differently, either when you declare the table during ingestion or when you change the table definition with DDL. Then the engine distributes tablets to the correct nodes so they are physically together.

What if you need to backfill new data? Backfilling is the same as updating data. Either you set up a Flink job that ingests older missing data slowly, one batch after another. Or you do an Iceberg export that you can load from. Fresha uses StarRocks pipes for this. Either way works.

Schema changes are a bit different from other systems. You need to stop ingestion from Flink, change the schema and dataset coming from the Flink job, change the DDL statement for your table definition in StarRocks, and then restart the pipeline. Only then do you see the new column.

Tradeoffs

Anton says these are always the same: you can have super-fast real-time ingestion, but then it’s more expensive. Or you have a little longer latency until everything is ingested, but at a lower cost.

Most important is the query speed, where their baseline is under 1 second (web analytics p95 is ~100ms) across all queries. For the business user, it should not matter whether joins are happening under the hood, whether data is stored in S3 with Iceberg, or whether it is coming directly from the StarRocks internal format.

If data is fully cold and on S3, StarRocks still manages 3-5 seconds of query latency based on extensive Puffin and Iceberg metadata sorted in the right order, Anton says.

Check out also the Fresha Data Engineering Blog for many more insights from the team about best practices around StarRocks and more. Or check out their Northstar utility for StarRocks, or the recent video about StarRocks at Fresha.

Pinterest: From Druid to StarRocks

Perhaps the biggest company that uses StarRocks is Pinterest. I wasn’t able to reach anyone to interview, but there’s a great article online at Delivering Faster Analytics. Let’s analyze it.

Pinterest migrated from Apache Druid to StarRocks to power their Partner Insights tool, which provides real-time analytics dashboards to advertisers tracking ad performance across 500+ million monthly active users. After the migration, they reduced p90 query latency by 50% while using only 32% of their previous infrastructure. This is roughly a 3x improvement in cost-performance efficiency.

Their setup runs on 70 backend engines and 11 frontend nodes. The MySQL compatibility allowed easy integration with existing tools, and StarRocks’ native ingestion eliminated the need for heavy MapReduce jobs in their data pipeline.

Additionally, there are some interesting comments on Reddit related to this article. People using Druid and ClickHouse stated that:

Both can handle multi-petabyte deployments the product seems to cover gaps within both architectures (Druid has an extremely heavy/complex footprint with limited join capabilities and is fairly costly to run). While Clickhouse can handle multi-petabyte volumes, certain design choices architecturally prevent it from auto rebalancing data which is critical at larger scale data volumes.

He would use StarRocks “largely because user requirements are only getting more complex at our company and architecturally/capability-wise StarRocks seems to be really the only OS solution that actually has near-direct compatibility with the MySQL protocol, materialized views and supports both real-time + batch loads, upserts. There is just a lot you can enable with just those four capabilities.”

Index Exchange & Others

It sounds similar from Ivan Torres, Staff Engineer at Index Exchange, who says:

I use StarRocks open source on K8s and benchmarked it against Druid. I also used ClickHouse before. For real time data they all come pretty close, some optimizations of each database work better for specific use cases, but you can achieve pretty similar numbers with the three of them.

What made me go to StarRocks is that it is a lot more versatile. It also does pretty good ad-hoc analytics and integrates with external catalogs like Iceberg for reading external tables. You can also have full separation of storage and compute and achieve good performance with disk caching.

He goes on to compare: in “Druid, joining tables ad-hoc is pretty much impossible”. And in ClickHouse, the “integration with external tables is not yet there,” he said.

But there are also some limitations. I had someone at a large payroll company reach out. They found it quite difficult to tune at scale and decided to migrate away from it. Difficult to manage, as in a full-time job, but powerful. They liked the StarRocks model, but didn’t have the engineering resources to maintain and tune it beyond what they could support at the time. They split workloads between ClickHouse and Snowflake based on use case.

As always, it’s a tradeoff. You need the right system for the right use case and the right engineering skills to manage it yourself, or just use the hosted version of each platform.

Common Patterns Across Adopters

Recapping before we go into the technical deep dive. Common patterns and use cases I found while interviewing the above companies and people:

• Joins as the really powerful part: Everyone mentioned the capabilities of joins as why they chose StarRocks over ClickHouse, Druid, or Pinot. Not just that joins work, but that they work without heavy pre-denormalization in Flink/Spark first.
• Faster time-to-baseline: Getting a working baseline to iterate from, compared to more complex setups, saving costs as the engineering team understands it faster.
• Use it when you want to speed up Snowflake or directly read from Iceberg tables: Limits are hit when customer-facing apps need sub-second query-response times.
• Hybrid hot/cold as the deployment pattern: Both Coinbase and Fresha run streaming into StarRocks for recent data while federating over Iceberg for cold historical data. StarRocks becomes the unified query layer across both.

Besides all the advantages, it’s clear that good data modeling is still required at every step. Colocated joins require upfront planning on how to set the same bucket key, count, and replica placement across tables. Basically, the partition key not only for a single table, but across a set of tables for best performance. Data flow and schema design should get a lot of love, as always in my opinion.

Now that we have heard a lot from actual users, let’s go into the details and analyze the parts we haven’t covered yet under the hood to see how the speed is possible and understand the architecture decisions that have been made.

StarRocks Technology Decisions and Architecture

Before we end, let’s look at some more of the interesting architectural decisions StarRocks makes.

Caching: The Alternative to Ingestion

In shared-data mode, data sits on slower object storage. StarRocks mitigates this with a multi-tier cache: memory -> local disk -> remote storage. Queries hit hot cache first, and cold data gets prefetched based on optimized strategies.

This means you don’t need ETL jobs to “load” data into StarRocks. You create an Iceberg catalog, point it at your existing tables, and queries automatically warm the cache over time. For predictable workloads, use the cache warmup command to proactively load specific tables before users hit them.

How it works under the hood: When querying Iceberg on S3, StarRocks splits remote Parquet/ORC files into fixed-size blocks (default 1MB) and caches them locally. Each block gets a unique key based on filename, modification time, and block ID. The first query fetches from remote storage, and subsequent queries read from local NVMe/SSD. Cached data persists across restarts.

On top of block cache, an in-memory page cache stores decompressed data pages and metadata. Hot data lives in memory, warm data on disk, cold data stays on S3. StarRocks also caches Iceberg metadata (manifests, schemas) to avoid catalog round-trips on every query.

The tradeoff is that the first queries on cold data still hit remote storage, but once warmed, performance matches native tables.

Real-Time Updates Without the Merge Overhead

Beyond caching, StarRocks handles real-time data updates efficiently through its columnar storage engine. Data of the same type is stored contiguously, enabling better compression and reduced I/O as you query only the columns you need.

But what makes it interesting for real-time analytics is how it handles updates. Traditional OLAP systems use different strategies:

StarRocks uses the delete-and-insert pattern. Instead of merge-on-read (which pays the merge cost at query time) or copy-on-write (which rewrites entire files on updates), StarRocks maintains a primary key index with delete bitmaps. Updates mark old rows as deleted and insert new ones. No expensive sort-merge at read time. This means sub-second data visibility for upserts while keeping query latency predictable, even on large update volumes.

The storage engine guarantees ACID for each ingestion operation: transactions either fully succeed or fail, with isolation between concurrent loads.

Cost-Based Optimizer (CBO): Avoiding Joins Without Upfront Denormalization?

On-the-fly joins aren’t typically a strength of OLAP databases, but they are one of StarRocks’ strengths, as we discussed. If your execution engine is fast enough, you avoid pre-building denormalized data marts, saving storage costs and ETL complexity.

Why are StarRocks joins fast?

The cost-based optimizer navigates the exponential search space of join plans, automatically transforms expensive join types into cheaper ones, and aggressively pushes predicates down before data reaches the join.

The optimizer leverages rich statistics including histograms for skewed data distributions and multi-column joint statistics to produce accurate cardinality estimates that guide join ordering and execution strategy selection.

When you still want denormalization: For high-concurrency scenarios serving hundreds of simultaneous users, pre-aggregated views still win. The difference is you start with normalized data, query it directly, and selectively add materialized views where needed. Fresha, NAVER, and others use this feature, e.g., NAVER achieved 6x speedups on specific high-traffic queries.

Intelligent Materialized Views

StarRocks reads Iceberg/Hive tables in-place without copying data. Its vectorized engine processes Parquet/ORC directly, and the only overhead is metadata lookup. No transformation is needed.

The query flow:

1. Frontend receives query → queries catalog for table metadata
2. Extracts file paths from Iceberg manifests (no data read yet)
3. Distributes file locations to Backend nodes
4. Backend opens Parquet/ORC directly from S3, applies predicate pushdown
5. Reads only required columns (late materialization)

On top of this, StarRocks’ intelligent materialized views auto-refresh based on base table changes and are selected automatically at query time. The optimizer rewrites queries to use MVs when beneficial. No manual intervention is needed.

This enables a layered approach to data modeling without traditional ETL pipelines:

Starting from the bottom: raw data sits in your data lake (Iceberg, Hudi, Delta Lake, Hive). An external catalog MV can transform this into normalized tables, still queryable for ad-hoc analysis and OLAP workloads. From there, async MVs can create denormalized tables for faster OLAP queries. For high-concurrency standard reports, aggregation MVs (roll-ups) pre-compute the heavy lifting.

The key insight is that you don’t build all these layers upfront. You start with normalized data, query it directly, and see if it’s fast enough. Then be selective and add MVs where you see bottlenecks. You preserve the single source of truth in Iceberg while progressively optimizing hot paths.

To repeat, at a high level, StarRocks achieves its performance through four architectural decisions: Colocate Join for zero network overhead on co-located data, delete-and-insert for O(1) updates instead of merge-on-read, direct Parquet/ORC reading with no ingestion transformation, and SIMD Vectorization for faster filtering/aggregation on Parquet computation to determine whether data rows are empty.

Conclusion: Is StarRocks Too Good to Be True?

So when should you actually choose StarRocks? After interviewing Coinbase, Fresha, and digging into Pinterest’s migration, the pattern is clear. Choose StarRocks when joins are central to your analytics.

ClickHouse excels at single-table aggregations and observability. But if you’re constantly pre-denormalizing in Flink or Spark just to avoid joins, StarRocks lets you skip that pain.

Here’s my mental model as of now:

Again, choose StarRocks when you hit cloud DWH speed limits for customer-facing analytics, need real-time analytics (<50ms), have frequent updates/deletes, want to query Iceberg/Hive directly, need complex joins, or want MVs on external tables.

But advanced features don’t come without tradeoffs. You have to choose a partition key across colocated tables, but you can only optimize for one of them. Overall, good data modeling still matters. And if one of them is your main use case, it’s always best to create a Proof of Concept where you test and compare the differences.

If you’re looking for a real-time BI and dashboarding tool designed to benefit from StarRocks’ real-time query performance, check out Rill and the recently released native StarRocks connector. With it, you can connect directly to your StarRocks instance and query data in real time without first ingesting it into Rill, expanding Rill’s support alongside ClickHouse, Druid, and Pinot. You simply specify StarRocks in their source YAML and off you go. Check out the docs for more information. All open-source if you want. Cloud-ready if you need.

Special thanks to Eric and Anton, who took the time to answer my questions and helped me learn a lot about how StarRocks works. Follow them on LinkedIn and subscribe to their blogs and posts.

References

1. Bajaj, K., Luo, Z., Yang, Y., Barai, S., & Hu, M.-M. (2024, July 31).
   Delivering Faster Analytics at Pinterest. Pinterest Engineering Blog.
   Link — Describes Pinterest’s migration from Druid to StarRocks for their
   Partner Insights platform, achieving 50% p90 latency reduction at
   32% of the previous instance count, resulting in a 3x cost-performance
   improvement.

2. Vuong, H., & Cao, H. N. (2025, March 6). Building a Spark observability
   product with StarRocks: Real-time and historical performance analysis.
   Grab Tech Blog.
   Link — Describes Grab’s “Iris” Spark observability platform redesign, migrating
   from a TIG stack (Telegraf/InfluxDB/Grafana) to a StarRocks-centered
   architecture to unify real-time + historical analysis and simplify
   ingestion/visualization.

3. Shekhawat, V., & Andrews, M. (n.d.). From BigQuery to Lakehouse: How
   We Built a Petabyte-Scale Data Analytics Platform – Part 1. TRM Blog.
   Link — Explains TRM Labs’ move from BigQuery + distributed Postgres toward a
   lakehouse architecture, selecting Apache Iceberg for table format and
   StarRocks as the query engine for low-latency, high-concurrency
   user-facing analytics.

4. Event Recap, StarRocks Singapore Meetup #2 @Shopee. (n.d.). 知乎专栏
   (Zhihu).
   Link — Event recap for a StarRocks community meetup hosted at Shopee’s
   Singapore office, describing talks and themes around customer-facing
   analytics use cases.

5. Shen, S., & Sun, E. (2024, June). Data Warehouse Performance on the Data Lakehouse [Lightning Talk]. Data+AI Summit 2024, Databricks. Link — A joint talk by CelerData and Coinbase presenting how StarRocks delivers data warehouse-level query performance directly on the data lakehouse.

Full article published at Rilldata.com - written as part of my services

The tools change. The loop doesn’t.

The Invisible Plumbers

When I started my career in 2003, there was no “data engineering”. There was no big data, no data science. We called it Business Intelligence. Data Warehouse Developer. ETL Developer.

We were the plumbers of the organization. And like plumbers, nobody noticed us until something broke.

Being a data engineer means: you’re building the foundation that everyone stands on, but when the presentation goes well, the data scientist, the app developer, anyone who presents gets the applause. When the executive makes the right decision, the analyst gets the credit. When the dashboard loads in 1 second instead of 20, nobody says anything at all.

But when one number is wrong? When a pipeline is 10 minutes late? When someone asks for “a small change” and you explain it’ll take a day, or a week to fix it?

That’s when everyone notices you. And shares their opinion on how to make it better.

“Why does this take so long? It’s just one column. Why isn’t it real-time?”

They don’t see the 147 downstream dependencies. The three systems that need a fuzzy-logic join. Or the security measures that go through three different subnetworks. The backfill that’ll take 6 hours to run. The schema that hasn’t been touched since 2021 because the last person who understood it left the company long ago.

This is the paradox of data engineering: when you do your job, you’re invisible. When anything goes wrong, you’re under a microscope.

The Epochs: A 50-Year Journey

To understand where we are today, you need to understand where we came from.

1970s: The Beginning

Edgar F. Codd proposed [[SQL]] in 1970. A way to abstract the complexities of data storage. By the 1980s, it became the standard. IBM built System R. Oracle launched their RDBMS in 1979.

The foundation was laid. But nobody called it “data engineering” yet.

1980s-1990s: The Warehouse Era

[[Bill Inmon]] formalized data warehousing principles in the 1980s. Many call him the father of data warehousing. Then in 1996, Ralph Kimball published “[[The Data Warehouse Toolkit (Ralph Kimball)|The Data Warehouse Toolkit]]” and gifted us with [[dimensional modeling]]—star schemas, fact tables, slowly changing dimensions.

These concepts? They’re still relevant today.

2000s: When “Big” Changed Everything

The dot-com bubble burst. Tech titans were born such as Google, Amazon, Yahoo, hitting walls their databases couldn’t scale past.

So Google released two [[Data Engineering Whitepapers|groundbreaking papers]]: the Google File System in 2003, MapReduce in 2004. Yahoo responded with Hadoop in 2006. Hardware prices plummeted.

Suddenly, we weren’t just BI engineers anymore. We were “Big Data Engineers”. We had to know traditional relational databases AND the new open-source filesystems. The skillset kept expanding—from data modeling to software development to mastering Hive and Spark, all coordinated with R and Python.

The term “big” was everywhere. But how big is “big”? Nobody really knew. We just knew the old ways weren’t working anymore. And Facebook and co showed us the way.

2010s: The Cloud Changes the Game

Amazon announced AWS. Google Cloud and Azure followed. Companies no longer needed to own hardware. The flexibility was unprecedented, and we could get any DWH on demand.

Redshift. Snowflake. And then the open-source wave hit:

• Airflow for orchestration (2014)
• Superset for visualization (2015)
• dbt for transformation (2016)

And in 2017, Maxime Beauchemin—after creating both Airflow and Superset—published “The Rise of the Data Engineer”. He defined, for the first time, what data engineering actually meant. He explained the shift from business intelligence to data engineering.

I remember releasing my first viral article in March 2018: “Data Engineering, the future of Data Warehousing?” It got 200 likes. Back then, that was a lot 😉.

Since then? New technologies appeared weekly. The [[Modern Data Stack]] was born.

2020s: DevOps Meets Data Engineering

This is where it gets interesting.

Data engineering isn’t just about moving data anymore. It’s about infrastructure as code, version control for data, CI/CD pipelines, Kubernetes, Docker, and Terraform.

The skills needed have exploded. You need to know:

• SQL (still the foundation)
• Python or Scala
• Cloud infrastructure (AWS/GCP/Azure)
• Linux and bash scripting
• Git for version control
• Data modeling (the lost art)
• Business logic (the most important)

DevOps principles are now [[The State of DevOps in Data Engineering|table stakes]]. You’re not just building pipelines. You’re building systems that need to self-heal, auto-scale, and deploy without downtime on any environment.

And today? AI agents? They’re the latest chapter. But under all the hype is the same eternal truth: you need fresh, organized, clean data.

The Eternal Loop: Same Problems, New Tools

Here’s the uncomfortable truth: we’ve been solving the same problems for 50 years.

In 2005, we had SSIS and star schemas. “The cube is rebuilding” was the pain point.

In 2015, we had Hadoop and Spark. “The cluster is full” was the nightmare.

In 2025, we have dbt and Snowflake. “The bill is how much?” is the new horror story.

The tools change. The problems don’t.

Last month I analyzed a 200-line dbt model as part of a larger GitHub repository. You know what it was doing? Exactly what we did in 2005 with stored procedures. Same business logic. Different syntax. I laughed. Then I cried a little. (just kidding, I didn’t 😆)

An old data warehouse architect from 2003 once drew a star schema on a whiteboard in 40 seconds. It would take my team three sprints to model in Oracle Warehouse Builder (OWB). He said, “We called it just another day at the office”.

We’re not really any smarter than the people before us. We just have better marketing 😉.

What Actually Matters (And What Doesn’t)

Here’s what I’ve learned after 20+ years.

The Excel File That Saved Me

I was in a coffee meeting with a finance analyst—call her Maria. Fifteen years at the company. She opened her laptop and showed me an Excel file (sometimes it was Microsoft Access DB with a custom UI!).

Forty-seven tabs. Formulas referencing other files on a shared drive. VBA macros from 2012. VLOOKUP nested inside SUMIF.

“This is how we calculate the quarterly forecast”.

From my perspective of making it available to everyone and needing to understand it, I was a little shocked. I’d spent three weeks reverse-engineering the business logic, trying to understand it, trying to recreate it in SQL Server, and adding it to our data warehouse. But the numbers were never the same, close most of the time.

After having multiple such experiences, sometimes Microsoft Access databases with custom UI built in (!!!), I learned something. Though my initial reaction was shock and horror, I learned that Excel isn’t the enemy. Excel is the business telling you what they actually need.

When someone asks to export to Excel, they’re not rejecting your work. They’re telling you something. Maybe your dashboard is too slow. Maybe they need to add a column you didn’t think of. Maybe they just need to feel in control of their analysis.

Power users will overengineer everything, but ask them for the Excel file and you might get validated business logic and ETL code for free. Win-win.

The Real-Time Lie

Everyone wants real-time. “We need to see this data instantly”, they say. “For decision-making”.

I always ask: “What decision will you make differently if you see it 10 minutes sooner?”

Most of the time, they can’t answer. There’s a small percentage that needs it: air traffic control, fraud detection, Black Friday e-commerce. But the rest? They just think real-time serves them better.

Real-time adds a much higher complexity. Harder debugging. Harder backfills. Harder historization. The question is, for what? So someone can watch a number update every 30 seconds instead of every hour?

Push back on “real-time”. Start with hourly refreshes. It’s almost always enough.

The Schema Change: A People Problem

They said it was small. Just renaming user_id to customer_id.

You trace the lineage. 147 downstream dependencies. Three teams. One undocumented view from 2019 that somehow powers the CEO’s dashboard.

That’s when you realize: schema changes are usually people problems, not technology problems. The reason things break is when upstream producers don’t own responsibility for downstream analytics and don’t communicate the changes. There’s no process in place. Just assumptions.

Fix the people process first, then update the code.

The Lost Art of Data Modeling

Max Beauchemin once said in an interview: “I like the analysis side. I think I’m a good data modeler. It’s kind of a lost art, so I still do a lot of our data pipelines”.

He’s right.

After years of “just dump it in the data lake”, people are rediscovering that structure matters. Data modeling forces you to think about:

• [[Granularity|Grain]]: What’s the lowest level of detail we need for this data?
• [[Entity Relationship Diagram (ERD)|Relationships]]: How do these entities connect?
• [[The Goal of Business Intelligence|What the business needs]]: What user insights they cannot know today, but lie in the source system provided, or combined with other data sources.

It’s the difference between a data warehouse and a data dump.

But here’s the thing: I believe AI will bring us back to the fundamentals. When AI-generated code breaks and you’re out of context, what then? That’s where the fundamentals save you. Data modeling. Understanding the grain. Knowing SQL deeply, not superficially¹.

Someone needs to understand and refactor generated code. Someone needs to simplify. That someone is you.

The Lost Code We Inherit

You’ll inherit code from someone who left. Everyone does.

I once found a DAG called final_v3_FIXED_REAL_FINAL.py. Inside was a comment:

1

# Mike: I don't know why this works. Just leave it

Mike was right. I left it.

The biggest pitfall? Trying to recreate everything to your taste. Accept the legacy. Adapt or improve one thing at a time. The motto “Don’t touch what works today” really applies to legacy code most often.

Usually, the previous engineer wasn’t naive or stupid. They were solving different problems with different constraints. Your job isn’t to make it beautiful (sometimes it helps!). Your job is to keep it running while slowly making it better over time.

The Books That Actually Matter

As the cycles come and go, these books helped me throughout the cycle², and can be applied to this day.

Designing Data-Intensive Applications by Martin Kleppmann about distributed systems and how to build them. Wait a little, version two is just around the corner.

[[The Data Warehouse Toolkit (Ralph Kimball)|The Data Warehouse Toolkit]] by Ralph Kimball. Someone in 2045 will still need to understand fact tables and dimensional tables.

Fundamentals of Data Engineering by Joe Reis and Matt Housley. A great start to know about all the concepts and principles you hear everywhere, including in this article.

Patterns of Data Engineering (PoDE) by me. If you want, you can also read my unfinished online book, which starts with the state of the art, the history and key convergent evolution in data engineering, about the ever-returning cycle we talk about here, and explains them with higher-level patterns.

The first two books don’t mention Snowflake, Lakehouse or dbt. They mention problems that existed in 1995 and will exist in 2045. That’s the Lindy Effect, and how you know they’re worth reading.

What I Know Now That I Wish I Knew Then

If I could go back to 2003 and talk to my younger self, here’s what I might say. Boy:

1. The tools will change. The fundamentals won’t.

Stop chasing every new framework. Learn [[data modeling]]. Learn how data is flowing. Learn SQL deeply, not superficially. Learn how humans make decisions. Everything else is syntax.

In 2026, AI helps us write code faster. But someone still needs to understand the fundamentals, the [[Data Engineering Lifecycle]]. That someone can be you.

2. Talk to the business people.

This is a crucial lesson in my journey. What you’ll learn from them will make you inevitably a better data engineer. Technical skills can be learned, outsourced, automated. Knowledge about the business is much harder.

The best data engineers aren’t the ones who know every new tool. They’re the ones who know why the data matters.

3. You’re building the foundation, not the showcase.

When the presentation goes well, the data scientist, the AI engineer gets the credits. When the executive makes the right decision, the analyst gets credit. When the dashboard loads fast, nobody says anything.

But when one number is wrong? Everyone sees you.

Accept this. You’re a plumber. Be the best plumber in the world and make sure nobody ever thinks about the pipes.

4. Data quality is learned through pain.

You can’t understand data quality from a textbook. You need to see bad data. If you start looking, it won’t take long, and you’ll see really bad, production-breaking data. That will teach you what good data looks like.

And you’ll only get faster by talking to the people who use it.

5. Presentation matters more than you think.

No matter how fancy your pipeline, how elegant your code, how profound your insights—if the presentation isn’t right or the data quality is terrible, no one cares.

Throughout my career, presenting data understandably has been as important as building the pipeline. That’s why these days, I focus on the storyline and craft extensively.

6. Set boundaries early.

This job will take everything you give it. The people who succeed aren’t the ones who work 80-hour weeks. Sure, in the beginning you need it here and there too. But over time, you need to learn to [[Hell Yeah or No|say no]]. Document things so you can take vacation. Build systems that don’t require you to be online at 3 AM.

Future you will thank you.

7. Don’t chase every trend.

Data engineering is still going strong. Stronger than ever. AI won’t take our jobs any time soon. The opposite is true. There will be more chaos, and people who know how to model data, understand business requirements, and deliver high-quality insights will always be needed.

Plus, every AI solution out there needs data, a lot of data, and probably a plumber to fix the pipeline too. Use the knowledge of past years building data pipelines. We don’t need to rebuild everything every 5 years.

The Loop Continues

It’s 2026. I’m building a pipeline with DuckDB and Rill. The business wants faster dashboards and better insights. They want to edit data themselves. They want to use an AI chatbot. Or sometimes they just rename a column in the source system without telling anyone.

Here we go again.

But here’s the thing: I still love it. Especially when I can write about the learnings.

I don’t miss the late nights or the schema changes or the never-ending rewrites. I love the moment when you finally get the data right and someone in finance sees something they’ve never seen before. When a dashboard actually changes a decision. When the CEO asks a question and you can answer it with data.

That’s the job. Not the tools. Not the frameworks. Not the buzzwords.

The moment when data helps a human make a better decision.

The Final Truth

The tools will change. The vendors will rise and fall. Snowflake will be replaced by something else. The latest new shiny tool will become the legacy tomorrow. AI agents will be the next big thing, and then something after that.

But someone, somewhere, will always need to:

• Understand the grain of a business
• Know why the numbers don’t match
• Explain to the CEO that the data they want doesn’t exist yet
• Debug why a pipeline broke at 2 AM
• Figure out why production data looks different from dev data

That someone is you.

You’re the invisible plumber. The unsung engineer. The person who makes sure the foundation doesn’t crumble while everyone else builds on top of it.

And honestly? It’s a pretty damn good job if you like to work quietly, helping a large part of the business.

Because 50 years from now, when we’re using tools we can’t even imagine today, someone will still be ingesting data, modeling it, transforming it, serving it. Someone will ask for a change. Something will break.

The loop continues. The problems remain. Only the tools change.

And that’s okay. Isn’t that somehow beautiful? Because beneath all the hype, all the new frameworks, all the promises of “this time it’s different”—there’s you, the data engineer 😉. Understanding the data. Knowing the business. Building the foundation.

That’s [[Data Engineering]].

[!tip] Inspiration

This piece was inspired by the confessional storytelling style of Diary of a CEO. If you enjoyed this format applied to data engineering, let me know—I’d love to hear your own stories from the field.