My goal is to offer a collection of wisdom, insights, and principles gathered over the years. Giving you an outside view from someone who has used Dagster since back in 2019, used it at enterprise scale but also for my hobby projects (e.g. real-estate project). The piece should give you a holistic view of Dagster’s place in the data ecosystem, how to deal with the complexity of data architecture and enterprises, and scaling your data jobs.

This article shows you how orchestrators such as Dagster are built for an open data platform that integrates the full data ecosystem, with the shift to data assets instead of DAGs, reducing complexity and applying data engineering best practices.

[!note] Definition of Almanack (also spelled “almanac”)
The term refers to a publication containing a variety of information on a dedicated topic. The modern usage of Almanack, particularly in the context of books like those by Charlie or Naval Ravikant, is often metaphorical. It suggests a collection of wisdom, insights, or principles gathered over time.

What is Dagster

In late 2018, on a co-working and co-living sabbatical in Bali, I was searching for something to bring the data warehouse out of the drag-and-drop world of SSIS and Oracle reporting and into a code-first, developer-friendly workflow. I looked at ODE, BiGenius, TimeXtender, and WhereScape, but found that none of them quite fit my open source and programmatic preferences, so I tried to build something myself but didn’t succeed. A year later, back at my 9-to-5 in Copenhagen, I heard Nick Schrock on the Data Engineering Podcast describing the motivation and story behind a Python framework called Dagster that did exactly that. I was hooked, and have used Dagster ever since.

Early Focus on Developer Friendliness

To understand the context of 2019, you must understand that back then, most ETL jobs were triggered with cron or bash scripts, and if there was an error, the only option was to re-run in the next nightly window where production wasn’t touched. Dagster, as explained by Nick in the podcast, focused on developer-friendliness, in particular for ETL developers back then, and that focus hasn’t changed today for data engineers.

So what is Dagster? The original idea, started in 2018 during a sabbatical after Nick worked at Facebook, came with this definition:

One of the goals of Dagster has been to provide a tool that removes the barrier between pipeline development and pipeline operation, but during this journey, he came to link the world of data processing with business processes.

Today the definition hasn’t changed much and reads like this from the Dagster Docs:

Dagster is a data orchestrator built for data engineers, with integrated lineage, observability, a declarative programming model, and best-in-class testability.

The initial definition to “link data processing with business” was the key reason that brought me to it, along with the quality of how the components were implemented. Even more compelling was Nick’s visionary outline for 3-5 years ahead: to make the work of data engineers similar to software engineers, and make their daily life easier.

Biggest Shift Early On

This vision led to many new concepts Dagster originally created, which we take for granted in today’s data work, and shifted the work into a more reliable and useful toolset for data engineers.

Data-aware Orchestration Shift

One of the biggest shifts compared to previous tools and orchestrators was that orchestration was fully data-aware from the very beginning. It tried to understand the heterogeneous complexity that exists at every small to large enterprise company, and thrive in it, supporting the full data engineering lifecycle with its platform and data pipeline capabilities built in.

This gave me a toolkit for building reliable data pipelines out of the box early on, with battle-tested features through its users (open-source) and a quality and thoughtfulness I hadn’t seen before. This was personified by Nick and could be vividly felt in the early interviews, but also in the code that the team produced openly on the repo.

For example, backfilling, restartability, or Spark integrations were open on GitHub, to use, and to adapt to your needs. It was a bit like dbt, but instead of modeling your SQL queries, you’d model your data pipelines and integrate complex data architecture.

This also meant that moving datasets or integrating dependencies is strongly supported, not an afterthought. In Dagster you can use resources to work with Polars, Pandas, Arrow, DuckDB, or anything else to pass datasets, and reference data assets declaratively, or even non-existing ones as Dagster knows to create the assets. Compared to Airflow, where you could load only small data with XCom for the longest time¹, this makes code simpler to understand and maintain.

Resources also decouple storage from compute. You could use Apache Spark locally with a single JAR file and in test use a full-blown cluster, or use MinIO as an S3 interface with simple bucket configuration and in production an S3 server from Amazon. Both are interchangeable without changing pipeline logic, by pure configuration — that’s the beauty of declarative data systems, and Dagster embraces this to this day.

Shift to Data-Asset Based Orchestration

Dagster was super early in shifting from DAGs and operational task-based orchestration to asset-based ones such as BI dashboards, tables, reports, ML models — artifacts the user actually cared about. Everything shifted from an imperative approach to a declarative one like Kubernetes or React, where you define what the dataset must or should have, and Dagster takes care of the implementation logic, mapping it to the configured filesystem, compute engine or cluster.

Now you could quickly describe each pipeline with declarative notations like update: daily or change to update: monthly, or if more advanced, you could define a small sensor logic that checks S3 for updates. Instead of implementing all logic in a data pipeline, we apply the logic directly to the asset and closer to the data, which makes it more transparent and integrated into the full data lineage. When updates happen, we have the full graph, but also a leaner and easier-to-maintain setup.

Or zoomed out - you see the focus on the assets, the tables themselves:

This led later to “Software-Defined Asset” and its mental model where you can define an asset pre-runtime and declare connections to upstream or downstream data assets (e.g. real-time housing prices that we fetch from a webpage that does not exist beforehand). Now we can already build and implement our graph and data lineage without having to physically create the dataset first. Software-defined assets use code to define the data assets and are version-controlled through git and inspectable via tooling. This transparency allows anyone in your organization to understand the canonical set of data assets and reproduce them at any time, and also lays the groundwork for asset-based orchestration.

As Rich Hickey said, the aesthetics of a programming language do not matter, only the outcome. Assets play into that fact. In data engineering terms, it’s not code but data pipelines and DAGs, but what everyone cares about are their outcomes: the data assets.

That led to the shift from working with tasks and DAGs to data assets. That’s the developer-friendliness built in from day one: develop locally and deploy to test and production, with infrastructure and technical implementation decoupled from business logic.

It made Dagster a bit more complex to start with — you need to know more upfront — but since every enterprise hits these data engineering challenges eventually, it’s better to embrace the fact and build for it. The result was an improved developer velocity, but what I noticed, too, was the joy of building reliable data pipelines, equipped with tools that helped me deal with errors, infrastructure, multi-tenancy, data science, big data, and everything thrown at me back then.

Not only single Purpose

When I first introduced Dagster at my previous company, all of a sudden other teams started to take notice and also wanted Dagster for other work such as provisioning infrastructure with one-click deployment. Especially the cloud platform team needed a tool to automate its scripts to deploy on Kubernetes, OpenShift, and everywhere else, but other teams also had needs to automate. With Python as the programming language for Dagster, reading from an external FTP server, transforming the data, and uploading it somewhere via API were not multi-month projects across different teams. Dagster’s flexibility and integration into other tools and systems were a key strength for most teams.

Observability and monitoring were another addition. Every run was logged in the UI and everyone could see the rich metadata of each pipeline run. And because it was open source and had a rapidly growing community, support and ideas didn’t run out.

Dealing with the Complexity of Enterprise Systems

If you have worked at any company larger than 10 people, you have noticed pretty fast that you are dealing with multiple source systems, different CRMs, different ERPs, multiple cloud platforms. Most enterprises have all major cloud platforms running in production, whether it is Amazon services, Google GCP, Azure, or any other major platform. You as the data engineer are the one making sure to integrate them, and basically deal with the complexity that comes with it.

How to Reduce Complexity?

First, acknowledge it: heterogeneous data complexity is a fact of the enterprise data lifecycle. Second, lean on tooling with technical integrations and written code that implements each vendor’s API, so we don’t build everything from scratch repeatedly. Third, work around the actual data assets the users want, not DAGs. With assets we declare outcomes, tests, and dependencies, and the system handles the rest. That cuts dependency hell, an unproductive Software Development Lifecycle, and the fear of change.

Composable is Making Systems Simpler

A more holistic framing comes from Simple Made Easy by Rich Hickey, creator of the Clojure functional programming language, where he debates what makes systems complex: state and objects, lots of vars, syntax, inconsistency. His conclusion is that composable is what makes systems simpler (like in music for a composer, which is what he created Clojure for): the ability to assemble, reassemble, and swap individual components into a flexible whole.

Dagster has exactly that ability, too. It integrates the core principles of Functional Data Engineering — pure and idempotent tasks, immutable partitions, reproducibility, versioning — directly into the framework.

State, Hickey points out, is never simple. Unfortunately for us, data engineering is all state: every datum is tied to a timestamp of when it was created, processed, or backfilled. Fortunately, helping us manage that state is what Dagster does: data assets, virtual assets, partitions, incremental materialization, and partitioned asset checks that evaluate a specific partition of an upstream asset instead of the whole dataset.

Rich concludes “Simplicity is a choice”, echoing Leonardo Da Vinci:

Simplicity is the ultimate sophistication

(Open) Data Platform

But how did the shift and the data engineering principles evolve, and how can we apply these as a unified solution?

With Dagster’s data-aware orchestration, shift to assets, separation of concerns, and multi-use, we are automating harder data and infrastructure deployment problems, and Dagster solves the problem of managing complex data environments more holistically. To me, it feels as though Dagster gave me this peace of mind and the toolset to simplify data engineering in a complex environment early on, and is the right tool for the ultimate sophistication for data work.

Dagster’s approach is composing a data orchestrator that integrates into any type of data work: from data integration with dlt, to transformation with dbt or just Python logic, to updating BI dashboards, to deploying on Kubernetes, all into a unified system. A fully open data platform, making orchestrating data and its flow simpler.

All of these features, combined with DevOps deployment strategies, make Dagster one of the data platform tools that has:

• An integrated UI and control plane for seeing what’s going on, unifying all your tools into a single webpage.
• Lets you see your data assets in a list with extensive metadata: it’s your data catalog showing all tables, BI dashboards, reports, and other data assets.
• Keeps and creates all the metadata when we run data pipelines across all data systems: With end-to-end access, we can also have metadata and data lineage end to end, which helps us understand where the data comes from, and in case of error, where the bad data is.
• Has integrated scheduling, sensors, backfills, and concepts to work with data, built in.
• Supports multi-team isolation through code locations, so different teams can own different parts of the platform without stepping on each other.

As Dagster is open source, it gets promoted from a usual data orchestrator to an open data platform, with the great advantage of transparency: easily patching an error or integration if you need to integrate an obscure system that only your company has, working on the cutting edge with the community, or getting features from them.

[!note] The Venn diagram of Dagster
Obviously you can’t optimize a data platform in all directions. If you look at Dagster as a Venn diagram, it has these three circles: the right abstraction, flexibility, and full automation through programmability.

Control Plane: Center with All Metadata

It integrates multiple different teams such as data engineers, platform and infra teams, with data science and business people who want to run their jobs. Feature-wise, it provides data catalog and contract capabilities, lets you see data assets you’re responsible for and when they last got updated, and shows their downstream and upstream dependencies, all in one real-time observability and monitoring UI.

This is all done through its control plane, which centers all metadata and unifies different data tools along the lifecycle into one platform, something usually only closed-source data platforms achieve. The control plane serves everything in a single view, showing how all processes are working. It’s the operational dashboard for your company.

With data orchestration as the heart of data work, with metadata for any process and access to all source systems we’re pulling from or intermediate systems, it’s in the perfect place to serve as the central metadata store. Think of INFORMATION_SCHEMA, but for overall data work, not only one single database. Only the orchestrator can understand the system and its status this deeply.

Why Open Data Platform: a System that Unifies Open Source

The open in open data platform is interesting, as it’s really hard to build a unifying layer across different stateful data systems, and it’s worth highlighting that Dagster achieved just that.

If we look at open data stack architecture: unlike cloud data platforms that have the same goal, such as Fabric, Snowflake, Databricks, etc., Dagster builds on open standards and is itself an open standard. It’s like a protocol in which we declaratively define our data assets (e.g. Software-Defined Assets, environments, resources) that then get automatically executed with composable computes we define in resources, all interchangeable. Even closed-source engines such as a Databricks Spark cluster work really well.

The hardest part is integrated data governance, lineage, access rights, and compute. An orchestration platform like Dagster doesn’t give you everything, but you get most of what you need, in an open and composable way.

Composable Data Stacks Possible

What this architecture allows is what Wes McKinney calls “Composable Data Stacks” in Monday Morning Data Chat, essentially rebundling the data platform.

Composable data stacks depend on compute engines. With this architecture, plus Dagster resources or external code (Dagster Pipes), we can easily pick and choose what is best suited for the task at hand, not only for different jobs but also depending on different environments. Although there will never be one singular tool for everything, it’s necessary that we have a layer of integration, and there’s no better place than the orchestration layer that separates execution and technical logic from business and already deals with multiple data environments.

Pete Hunt, the CEO of Dagster, said recently:

Our goal is to make AI as lightweight, accessible and cheap as we can to drive enterprise value, while enabling our customers with the infrastructure they need underneath - the orchestration and data platform layer where reliability, support, and recovery actually matter.

Pete is reaffirming that the data platform layer is the priority, so AI can build on top of a great foundation. In the eyes of a data engineer, it’s a dream come true to have an open platform that is declarative to simplify the overall architecture, but also uses AI for the glue code, based on best practices enforced through Dagster as the open data platform.

The Right Abstraction Layer for an Open Data Platform

Charlie Munger’s Almanack distilled decades of investing wisdom into timeless principles and mental models that compound over time. Here I tried the same for Dagster.

The principles touched on won’t be obsolete next year. Data-aware orchestration, declarative assets over imperative DAGs, separation of business logic from infrastructure, and composable stacks with a single control plane are all mental models for building data platforms that hold up whether you’re running DuckDB on a laptop or Spark across three cloud providers.

Eight years after discovering Dagster on a podcast during my time in Copenhagen, I’m still reaching for it whenever a system gets complex enough to need real orchestration. With extensive built-in data quality checks, best practices like unit-testing, local development to prod, separation of business and technical logic, code locations, Domain Specific Languages (DSLs) for non-technical people, pipes and components to run something in Rust or Go, and many more, the Dagster data platform gives you huge leverage building from strong foundations, with the flexibility to change along the way.

It’s the abstraction layer for data engineering to solve hard business problems, an open data platform with opinionated design decisions that compound the longer you build on them.

In the next piece, I’ll get into what it actually takes to operate this — architecture, deployment, and governance — as a follow-on to these principles.

Next Steps

Find Dagster’s official skills for the latest and most updated way of working with Dagster, to feed to your AI agent. Or read the blog post with more information. If using Airflow, migrate from Airflow, or use Airlift for an integration for legacy and critical DAGs still in Airflow.

Find awesome-dagster, and check out further readings of mine at Data Integration as Code: Configuring Airbyte and dbt with Python (Dagster) or Data Orchestration Trends: The Shift From Data Pipelines to Data Products.

Want to use all of this stress-free without the deployment burden? Use dagster+. Great tradeoff between cloud and OSS, still having the OSS Dagster foundation, but profiting from extra features (GitHub integration, cloning, etc.) and not needing to set up a DevOps pipeline or fiddle with Kubernetes.

Full article published at Dagster.io - written as part of my services

Chris has seen the data stack evolve over the years. He thinks AI will soon handle the majority of data engineering work, provided with the right tooling and access to CLIs and APIs. He also thinks LLMs might not speak SQL, but a format that represents data transformations. With so much shifting and changing currently in the AI space, new models, new workflows weekly, Chris’s perspective helps you navigate without overreacting, based on a long experience in the domain.

The article is structured in four parts: (1) correctness when working with financial data, (2) the Ralph Loop and why AI might be better off speaking something other than SQL, (3) vulnerabilities and the case for “Okta for Agents,” and (4) the future of AI — including why “data engineer” as a distinct role might not survive.

Introducing the Guest: #2 Chris Riccomini

Chris Riccomini is a Software Engineer, Author, Investor, and Advisor. Previously at WePay, LinkedIn, PayPal, and author of The Missing README: A Guide for the New Software Engineer and co-author of 2nd version of the iconic Designing Data-Intensive Applications book.

Chris has been working in open source throughout his career. He is the author of Apache Samza, a distributed stream processing framework. His current project is SlateDB, an embedded key-value store built on object storage. He is also on the Apache Airflow’s PMC.

Correctness of Data in the Financial Sector: How Does This Work with AI?

Chris had worked at financial companies where data correctness is essential. My first question was “How do you see using AI in data when financial services, or most other places, must be correct? How do you mitigate the small errors AI still makes in such a situation?” His response:

It really depends on where in the stack AI is being deployed.

Use Cases with Different Risk Profiles

Risk, fraud and compliance. The bar is model explainability, you need to know why the model made the decision it did:

If AI is involved in decisioning around risk and fraud, compliance and “model explainability” comes into play (why the model made the decision it did). This is one of the reasons we really liked random forest models at WePay: you could explain the actual rules that the model had derived and used in order to make a decision.

The data engineering context, compared to a traditional data modeling situation, is interesting:

If AI is being used in a data engineering context, it seems to me more like a traditional data modeling situation. You should be able to define invariants that must always be true for your data. For example, the ledger should always sum up. This is how we managed our data pipelines. If AI is defining data integration pipelines and moving data, the invariants should still hold. Traditional data verification tools will continue to play a role there.

For data analytics, this is where most of the fear lives:

There is a fear that AI will hallucinate and cause a bad decision to be made. I think this is a reasonable fear, but it’s also a problem we had before AI. Data in any organization is messy. Semantics aren’t always clear, contracts get broken, and so on.* *Every company I’ve worked for has had this problem. It’s not uncommon to find an incorrect query that’s been rolled up into a weekly ops review with the CEO, for example. This was true before AI.

So the question, is whether AI makes this worse or better. Chris own view has shifted recently:

If you’d asked me two years ago, I would have said it was definitely going to get worse. Now, I think it might actually get better, especially if we pair AI with a human. The latest LLMs have gotten really good at spotting bugs, inconsistencies, and so on. My personal experience is that I’m both more productive and more accurate with an AI.

I am having a similar experience: for working data engineering projects, if I use it for a not-too-distant future, meaning if the scope is clear and in a framework or rigid structure, it can implement a great solution since last December 2025, when the models got better. With it, it can go a long way, but still, it can’t work autonomously, or do a full project from scratch. It still needs a lot of hand-holding, as it does not understand the business.

So, balancing quantity with quality and keeping up with reviews at the speed of generation is also a challenge, especially since the model usually generates many lines of code. But for my writing process, where my personal voice plays a bigger role, I find that AI can’t help me too much yet in the actual writing process - but on the surrounding tasks (research, brainstorming, though also limited for new topics that are not based on existing ideas).

LLM Should Speak Substrait, not SQL

Chris said recently that: “Similar to my belief that LLM should speak substrait, not SQL”. I asked him to explain this quote and he said:

This is more of an intuition than something I’ve demonstrated to be true. But if you look at the way we use SQL, it’s actually used in two different ways: by humans and by machines. I think both can benefit from Substrait (or some equivalent).

Chris continues to explain that “Substrait is a format that represents data transformations. It has many operations that SQL has, but unlike SQL, which is purely logical, Substrait lets you define physical operations as well. In SQL, you say JOIN, but in Substrait you can say how to join: merge join or hash join? For those with a compilers background, Substrait can express both abstract and concrete syntax trees–intermediate representations (IRs).”

This is valuable for LLMs for two reasons:

1. You should be able to express SQL with fewer tokens (provided the serialization format for the logical operations is more efficient than english). This should make LLMs slightly cheaper to use, but more importantly it should keep them from hallucinating quite as much. (Granted hallucinations are less of a problem than they used to be).

2. More importantly, LLMs are pretty smart. They should be able to do query optimization really well. And Substrait gives them that ability–they can express physical operators (e.g. merge vs. hash), not just logical ones. This should allow them to do query optimization on the client side, and pass a physical query plan directly to the DB for execution (provided they have access to the requisite table statistics).

Substrait, as an emerging standard that provides cross-language serialization for relational algebra, is very interesting and something I want to check out, especially the expressiveness compared to SQL.

[!note] Downside of Substrait: LLMs are less familiar
Of course, there is a ton of SQL on the internet, so it’s not clear that LLMs will be as amenable to working with lesser known formats like Substrait. I think it’s worth experimenting with, though.

Making AI Output More Reliable

What I learned is that the longer something is in the future, the more vague or incorrect or hallucinated the outcome can be. So the more context and code you can provide, the more accurate the result. Which is pretty much in line with Substrait.

But how do we work with the LLMs, what’s the best approach, using god mode in OpenClaw or --dangerously-skip-permissions in Claude Code with no limits where it can go indefinitely with not much more context? I asked Chris if that’s also what he observed, and if he uses plan mode and a declarative approach or pipelines, as it helps for context and collaborating with the AI on a shared output, usually Markdown.

I was having coffee with a friend of mine, lamenting about this very problem a month or two ago. I was trying to get Codex to do something complex and it just kept falling on its face. My friend told me that you have to live in plan mode all the time. You can’t just ask it to plan the work, then flip to “Implement this plan.” You need to have the LLM iterate on the plan for many iterations. Probe its plan, ask it for details, ask it to expand sections, and so on. You need to get to the point where you feel like there’s no possible way the LLM can’t implement the plan incorrectly.

[!info] Spec-driven AI work

On the note of “needing to have the LLM iterate on the plan for many iterations”, Mark Freeman suggested in previous interview the spec-driven development (SDD) approach with the open-source GitHub Spec Kit, check it out or read the previous interview for more context.

The Ralph Loop: And Managing Context

After having a plan at hand, the next step is to keep the LLM’s working memory lean:

Once you have a good plan, you need to manage context. In some cases, you will need to take your plan and start with a fresh context in the LLM. In other cases, you’ll need to clear the context periodically throughout the work. I use a Ralph Loop for such cases¹.

I had the exact same experience when working with smaller code bases: to refresh context, the insights you gain over the iterations are not as effective if you add them bit by bit, compared to if you refresh memory and start over with all the new key insights provided at the very beginning, steering the model to a more tailored direction earlier on.

But with the Ralph Loop, which refers to understanding AI beyond surface-level applications, you get new insights that you can then add to your initial prompt, that you wouldn’t have gained otherwise, by exploring deeper programmable patterns.

The loop is an iterative, autonomous AI development technique where a bash loop (or plugin) repeatedly prompts an AI agent with the same goal, forcing it to persistently iterate until tasks pass external tests. It forces the AI to work, fail, and fix errors until success, rather than relying on the AI to decide it is finished.

On top of that, Chris says “You also need to impose a lot of quality gates. As with plan mode, you need to overdo it. ‘Quality’ is a bit of a squishy term’”, and he breaks it into three steps:

1. Define what quality is for your use case.
2. Measure the quality.
3. Enforce thresholds (gates) that your LLM must adhere to.

[!example] Example by Chris with the three steps for assessing quality

For example, part of your definition of quality might be test coverage. So that’s step 1. Then you set up a coverage tool for your codebase–step 2. Then, you put the phrase, “You always run tests before commit and keep test coverage above 90%,” in your CLAUDE.md. Finally, you install a git commit hook that enforces this rule.

This is a very rudimentary example, but you get the idea. There are a ton of different things you can measure and monitor for your work. I enumerate many in the post Code Quality Gates for Vibe-Coded Projects.

This essentially means we as the Prompt Engineers need to make sure that the workflow is correct, that we understand what we need to do, and accordingly adapt the workflow to get better code quality.

What about Functional Data Engineering, and Executing Deterministically?

In related terms, just as AI might hallucinate, it also might generate different outcomes with the same questions and same context. It’s non-deterministic. But data engineering works especially well if it’s done reproducibly, so we can backfill our data pipelines reliably and trust they will fill the same way.

This also ties into functional data engineering, running jobs with reproducibility and idempotent. I asked Chris what he thinks about this dilemma.

I’m not as worried about this as I used to be. A lot of tooling has popped up or evolved to help address this. Durable execution frameworks try to address some of this by papering over the non-determinism to keep replays deterministic by skipping the previously-successful parts of the flow. Ditto for traditional workflow orchestration systems like Airflow, Prefect, and Dagster. (Disclaimer: I have some Prefect shares.)

Moving to Incremental-loads for Better Determinism?

What I found interesting was Chris’s next suggestion: moving to smaller data sizes, and therefore to loading incrementally for a more reproducible outcome.

We can also move from full batch data processing to incremental batch data processing to help eschew some non-determinism.

A concrete example, splitting load by day:

Imagine, you have a bulk load job that always loads a full table from PostgreSQL into Snowflake, and that job does some LLM-based processing. Every time you re-run it, you’re going to get non-deterministic output. But if you convert it to an incremental job that runs daily and always loads the previous day’s data, then a re-run will only introduce non-determinism into the last day’s load. And presumably you’re re-running that day because something went wrong. In such a case, non-determinism is likely acceptable.

This is great thinking and shows it’s all about the use case and the risk appetite. If you have a lot less back reloads daily, compared to a full load, the accepted risk of one day might be acceptable, if you get great insights from the LLM, or something you’d need to do manually and then the alternative would be you either don’t do it at all, or very late when the insight is “less” valuable.

Side note, the engineering implementation of incremental loads might be much higher than a full load, as you need to add clear state management, checking what has run, and manage that state yourself, versus just running all. But this point almost certainly comes up in any case, whether you use AI or not, so we can factor out that fact in this scenario.

How to Prevent Vulnerabilities, and Work Securely with AI Agents?

Another hot topic with agents is security concerns around vulnerabilities. I asked Chris how he sees that domain in combination with generative AI, and also if we need “Okta for Agents”, as Maxime Beauchemin called it.

His view splits cleanly in two:

On the one hand, it’s a nightmare to manage these agents in the enterprise. On the other hand, they’re phenomenal at detecting compliance violations: leaked credentials, leaked PII, and so on.

He’d been thinking about an Okta-for-agents independently:

It’s funny you mention Maxime’s “Okta for agents” comment. I didn’t see it, but I’ve been saying the exact same thing. It seems patently obvious to me. What’s unclear is whether Okta is Okta for agents, or whether another company (or companies) will take its place. Innovator’s dilemma and all. Okta’s certainly give it a good try–their homepage is covered in it now.

Skills, Marketplaces and MCPs

He continues and says that it’s the wild west right now. You can load skills and even arbitrary skills from a marketplace and load any kind of text files without knowing if there’s a vulnerability.

There are examples where hidden code injection is done in a repo:

Chris continues with not having enough guardrails:

But yes, we absolutely need lineage, auditability, RBAC, ABAC, and so on. It’s the wild west right now (as far as I know, anyway). This is one of the reasons I was so outspoken about MCP when it first came out. I was very disappointed in their (lack of) security model. It’s the most important part, and it was completely lacking. It was rather shocking to me given Anthropic’s focus on the enterprise. More recently, they’ve added better support, though, so credit where credit is due.

Future with AI Agents

When asked about the future of AI, especially when we talk about data engineering, we discussed three interesting topics on what agents are doing well today, the role of data engineering itself and what programming language to use.

What Agents Already Do Well Today

I asked if we get self-healing data pipelines, so we do not need to get up at night, meaning AI does not only detect errors, but also analyses, debugs, pushes a commit to the repo and re-runs the pipeline autonomously?

I’ll be frank: I think AI will do the majority of the data engineering work in the future. I think we’re already at a point where it can; the tooling and practices just haven’t yet adapted.

This is an interesting point regarding tooling (and practices) not being adapted yet. Jeff Dean, Chief Scientist at Google DeepMind, made the point that Amdahl’s Law still applies, and that we need to re-engineer our tools as they were designed for human speed. If AI agents can run 50x faster, but the tools don’t, then we do not get an overall improvement.

On the other hand, what agents already do well today:

Agents are already excellent at inspecting failed Github actions, failed workflows, running SQL queries, writing Python–all the things data engineers do. As they get plugged into monitoring systems and begin to auto-remediate, the grunt work of data engineering will get taken over by AI.

And building new pipelines, given the right access:

Agents are also fully capable of adding new data pipelines, provided they have access to infrastructure to do so. If you stand up a fresh Airflow and add connections for all your systems, I’d wager an Agent can set up as many pipelines as you need on it. And if you define the security and compliance policies it should follow, it’ll do so.

Here, in my opinion, it is key that we use declarative and config-driven stacks, like Kubernetes and React are doing, and most modern tooling.

Data Engineering Role Going Away, or Unified?

Continuing on the thread of the future of AI, Chris talks about how shifting left is a movement we had for a while, and where this leaves data engineers as a role:

I’m not sure where that leaves data engineers. The “shift left” movement has been going on for a while. I can imagine a world in which “data engineer” as a distinct role goes away, or is folded back into a more generic data role that includes data engineering, machine learning, data analysis, and so on.

He’s been pushing this for quite some time:

We over-specialized the data space. It might have been necessary, but it isn’t now. So perhaps we’ll see “data” be a single role that encompasses not just data engineering, but analysis and machine learning/AI as well. I think that would be healthy.

Should We Let the AI Agent Choose the Language?

We heard people saying (e.g. Wes McKinney) that they choose programming languages, in this case Go over Python, based on AI, not what the human prefers. He calls it From Human Ergonomics to Agent Ergonomics. That Wes, the creator of Pandas and author of Python for Data Analysis (stay tuned, he will be the next guest for this interview series), chose Go is interesting, and is because its advantages in fast compile-test cycles and painless software distribution are key. Don’t worry, Python will not go away².

Or Ladybird is rewriting part of the browser entirely from scratch in Rust with agents in two weeks. So Chris, do you think that choosing the programming language will depend on the ergonomics of the agents in the future (or now already)?

In a word: yes. I have been pretty enthralled with the software factory concept lately. It’s how I do a lot of my development now. In that world, I just don’t care about the language my software is written in.

What he optimises for instead:

I care more about the characteristics of the output: its performance, stability, and cost to build (i.e. tokens). Languages that lend themselves to faster, cheaper, more stable LLM output are going to win.

These are very interesting thoughts, and I did a project fully vibe coded in Go to experience the cost-as-tokens as well. The codebase kept being small (apart from the tests), and therefore I could go much further with the given tokens compared to other projects where I used the same Claude Plan Pro and ran out.

Go is a language I don’t usually program in. And it is quite astonishing how far you get, but I also noticed a limitation as Lines of Code and size of the project grew, especially when adding new features that would break working features.

[!note] Side note by Chris: No more proofs for a programming language required

I used to think that this would pull us in the direction of languages that have formal methods properties. Proofs, model checkers, and so on. I no longer think that’s the case, though. I think LLMs have gotten good enough at writing code that proofs are no longer required. We can use normal testing strategies. I wrote more about this in The Waymo Rule for AI-Generated Code.

Does AI Take away the Learnings?

Last question I asked Chris — the danger of not learning new things, and getting overwhelmed with constant stimulation, and even addicted? In a world where we only prompt, where we don’t experience hitting a wall and then figuring it out, does that prevent us from learning new things? Are we just cruising on auto-pilot?

Chris mentions that it depends on how we use it and brings an example:

One could argue a calculator makes us learn less math; indeed, I keep an eye on that with my middle school-aged kids. But it’s also a tool that lets us do far more complex math without worrying about carrying the one or shifting the decimal, so to speak.

But you can also learn with AI he argues:

I have had instances where I learn a ton from AI. A concrete example: SlateDB’s language bindings. I built them all from scratch (or rather, AI generated them all from scratch). When I started, I knew nothing about bindings. As I worked with AI to steer it and iterate on the code, I learned about cbindgen, UniFFI, foreign function interfaces (FFIs), and so on. It’s a phenomenal tool for picking up something from scratch. I can ask it questions, learn from it, and so on.

Again, did he actually learn as much (from scratch, with AI) as he would have building it himself?

Almost certainly not, I think I would have learned a lot more [without AI]. But I also wouldn’t have done the work. Writing four bindings (Node, Java, Python, and Go) from scratch is just too much work. I don’t have the time for it. Especially since I have never written a line of Go, and I know next to nothing about the Node ecosystem. So in the real world, I think I came out ahead.*

Do We Learn Fewer Things?

Let’s finish with a question: Are we learning fewer or just different things? Something I’ve wrestled with for a while. Chris’s answer is:

Perhaps the things we are no longer learning don’t really matter anymore. Going back to the calculator example, I couldn’t really tell you in detail how a calculator physically works. If you took it apart and showed me its circuitry, I’d be unable to tell you anything about it, really. Does that matter? I’m not so sure.

I think we all are in this experience together, and nobody can really predict the future. I experienced both sides: when I rely too much on the assistant, I get more lazy and do the deep thinking less. While I course-corrected, and only used it for dedicated tasks, I noticed that abilities were improving again, or better, my feel and gut feeling got better again, and I had more confidence in the task at hand. But also, as Chris said, if I know it’s going to be a hard task, I can do much more because I deliberately use AI for certain tasks to actually finish the task. So the future will tell.

Next Interview

I hope you enjoyed this interview with Chris. Huge thanks to Chris for taking the time to speak with me and for sharing his experience with all of us. Follow him on LinkedIn, X/Twitter or on Bluesky, read his two amazing books. Follow his amazing newsletter, the new one at Posts on engineering, venture capital, AI, and more. | rng.md, but also his old one Materialized View has a wealth of insights.

There are three more interviews already lined up with great guests, one of them is Wes McKinney as mentioned, so please share feedback, questions you might want to ask or just your experience on how to work with AI in the data space. We’re all in this together, figuring it all out. The more we can learn from each other, what’s important, and maybe also what’s not, the better.

So stay tuned for the next interview.

Full article published at MotherDuck.com - written as part of my services

The core question is: “When should you store data internally in your warehouse versus externally in object storage?”. Hot data queried frequently goes inside. Cold archival data stays external, where it’s cheaper but slower. Interestingly, Databricks and BigQuery recently added external table features, but why? Not because they’re trendy, but because the economics still work.

This article offers an inside look at external tables, their 25-year history, how they evolved from CSV parsers to ACID lakehouse tables, and whether you need to know about them today.

What Are External Tables?

So what are external tables, and why have we been using them for so long? Why don’t we just use the internal storage of a database?

In Oracle, where I first used them in 2008, they allowed you — and still do — to access data in external tables. External tables are defined as tables that do not reside in the database, and can be in any format for which an access driver¹ is provided. All of this is provided via DDL (Data Definition Language) of the database, describing an external table with all its columns, data types, etc., exposing the data as if it were residing in a regular database table.

The external data can be queried in parallel and queried directly using SQL. Essentially, it’s read-only access to data stored outside of our database, making it available in a tabular, easy-to-work-with format to interact with existing tooling and language. In 2008, this was through procedural language such as PL-SQL in Oracle or T-SQL on MSSQL.

Today, external tables have evolved. The biggest change is that they can read more formats including semi-structured data such as Parquet, JSON, Avro, and ORC. While CSV was readable in 2008, the difference today is the columnar formats and nested formats that enable faster analytics. These are available for downstream processes and dashboards, but mostly accessed through SQL queries in one form or another.

A modern definition by BigLake, an evolution of BigQuery toward a multi-cloud lakehouse that tries to solve key customer requirements around the unification of data lake and enterprise data warehousing workloads, introducing external tables in 2015 as part of it²:

External tables are stored outside of BigQuery storage and refer to data that’s stored outside of BigQuery. [..] Google Non-BigLake external tables let you query structured data in external data stores. To query a non-BigLake external table, you must have permissions to both the external table and the external data source.

Snowflake defines them as:

[…] When queried, an external table reads data from a set of one or more files in a specified external stage, and then outputs the data in a single VARIANT column. Additional columns can be defined, with each column definition consisting of a name, data type, and optionally whether the column requires a value (NOT NULL) or has any referential integrity constraints.

External tables were added in 2021, and Snowflake described their benefits as follows:

External Tables Address Key Data Lake Challenges:

1. To augment an existing data lake. [..] augment their existing data lake, rather than replace it. The External Tables feature enables that use case. Customers can use external tables to query the data in their data lake without ingesting it into Snowflake. (side note: MVs³)
2. Ad-hoc analytics. Customers often use external tables to run ad-hoc queries directly on raw data before ingesting the data into Snowflake. Ad-hoc queries help them evaluate data sets and determine further actions.

Just a Pointer (Symlink)?

A simple analogy is a symlink in Linux, where you point from your current directory to another directory without moving data. You just add a pointer. If you read that file from that symlink, all it does is read it from the location the symlink points to.

An external table is the same, just a pointer to external data, bringing that data into the current data warehouse or cloud solution, hence the word external. You define the source format such as XML, CSV, etc., and define their structure, and then you can query that at any time. It’s similar to a SQL View in that sense, but pointing to non-internal data.

Running DROP TABLE and deleting an external table is metadata-based only. No data is removed, only the table definition from the internal data catalog. The same is true with a symlink. Almost any relational database today has support for it, even if it’s not called an external table. Everyone occasionally needs to read data outside of its warehouse or database.

Recap in the History of External Tables

Looking back at the history and evolution of external tables, we can quickly see that there’s a long history and they’ve been a recurring pattern across every generation of database technology since the early 2000s, and arguably longer if you count IBM’s federated database concepts from the late 1990s.

The Origin Story: ISO in 2001

The history starts with ISO/IEC 9075-9, published in 2001. Part 9 of the SQL standard defined foreign-data wrappers and datalink types for managing external data from within SQL. The work was completed in late 2000 and published alongside SQL:1999, with full integration in SQL:2003 (it was later updated in 2023).

It was the initial definition and extensions to database language SQL to support management of external data through the use of foreign-data wrappers and datalink types.

My first encounter was with Oracle external tables, but according to Wikipedia there were earlier implementations, such as Microsoft Access linked tables (~1992). Microsoft Access linked tables (~1992) were the earliest consumer-facing implementation where users could link dBASE, Paradox, text files, and ODBC sources as if they were Access tables. ODBC 1.0 (1992) itself established the first standard for heterogeneous data access across databases, though it didn’t create table abstractions.

Further, IBM’s DB2 DataJoiner (~1995) was more ambitious with a middleware product enabling SQL queries across Oracle, Sybase, SQL Server, Informix, Teradata, and even VSAM files through a unified interface. With SQL Server 7.0’s Linked Servers (1998) we got federated querying to Microsoft’s ecosystem via OLE DB, supporting cross-database joins with four-part naming conventions.

Most of these implementations shared a common limitation that Oracle (9i Release 1 - 9.0.1 in 2001) solved: they focused on querying other databases or required middleware. Oracle’s abstraction treated local flat files as first-class read-only table objects using the familiar CREATE TABLE ... ORGANIZATION EXTERNAL DDL syntax, providing a simple way to define external files as part of normal table creation and allowing ORACLE_LOADER access to query flat files (CSV, fixed-width, delimited) through DBAs.

It was an early way of separating declaration from compute (the Oracle loaders).

Why External Tables? What Are Their Benefits?

But why use external tables? What makes them so useful that they persisted? Why have they survived so long, and why are they getting added to Databricks and other major platforms?

For that, we need to look at external tables’ benefits. The first reason is that external tables can simplify data access to avoid developing ETL pipelines, moving data out of the source, and re-ingesting it in our data warehouse. They make external data accessible easily, defined in a tabular form by a database schema with column types. Typical cloud data warehouses like Snowflake and Azure use them to link existing data from object storage easily without moving data. This makes the object storage files accessible for almost any downstream tool or query language in a simple and cost-effective way.

Other ways of using them are to store some data on cheaper storage (e.g., object storage over data warehouse storage) and only link them in. It’s slower to fetch, but more affordable to keep. If you have large data sets, cost savings can be immense as this article shows, bringing down Snowflake internal storage cost from ~$23/TB/month to S3 infrequent access with ~$12.50/TB or S3 Glacier Deep Archive with only ~$1/TB.

Another handy side effect as the consumer of external table data is that the data is always up to date, because no refresh or update is needed. It goes without saying that this has its own downsides and can be a problem for the owner of the data if it’s used in production and the ETL process reads large amounts of data through external tables. This will affect upstream apps running or owning this data.

That’s why many use external tables in combination with materialized views (MVs) to truncate and recreate a daily snapshot (or similar) during off-peak (mostly nights) of this data, avoiding affecting production data and even optimizing query performance with added indices for downstream queries.

When Internal and When External Data? What’s the Limit of External?

The tradeoffs come down to how often the data is queried, e.g. the hot versus cold question.

The tradeoffs and considerations you should make when wanting to use them come down to the decision of how often the data is queried. The table below shows it in more detail:

In the era of data lake and lakehouse architectures, this is an important consideration. VSCO says: “disk space was growing more quickly than our compute needs,” which is what triggered the adoption of external tables.

If you look at your use case, if you need to do analytics across various sources with joins and augmentation of your data at an enterprise, you probably want to focus on loading data into your database or data warehouse, an architectural pattern that has survived more than 30 years. But if you have data that is external and small but you want to join it with existing data, or you always need fresh data and can live with a slower response time (maybe because it runs during the night), you might use external tables.

In any case, external tables are a good approach to keep in mind and a valuable toolkit to have.

They Work Well with Existing Tech and Common Patterns

Obviously, today’s external tables are not the same as the earliest ones in Microsoft Access, but the principle of accessing data outside your system is still the same. Nowadays we have more support, new formats besides CSV and JSON. We can do Parquet or open table formats.

As mentioned, they work well with related long-lasting data warehouse patterns and applications such as materialized views and stored procedures. The recurring pattern is to access external data with your data management system, similar to the pattern of materialized views that refresh complex SQL statements and make them fast, and stored procedures that run glue code within your database.

Moreover, there are temporary tables that are similar but only available during a transaction or session. They all work in the same Lindy effect, e.g., Databricks just announced Temporary table support recently on December 9th, 2025, or Databricks SQL Stored Procedure a little earlier, August 14th, 2025, for reusing existing SQL statements.

Again and again, everything that is old will be new again. Exactly what the Lindy Effect is all about. We can clearly say that the Lindy effect over the last 33 years applies here. The longer something is in place, the more likely it is to be around for at least that long.

[!info] External vs. Temporary Table

In contrast: temp table = session-scoped, writable, fast, invisible to others, auto-dropped. External table = persistent metadata, read-only, infinite size, visible to all, optimized for cost.

A common chain in practice is going from: external table → temp/transient table → permanent managed table.

How a Classical External Table Works

To understand how traditional external tables work, let’s first look at Oracle, which has built an extensive syntax around them and where they still work this way today.

First, we can create a place for external data called DIRECTORIES, which is simply a pointer or alias to a file system location where external files already exist:

1
2

CREATE OR REPLACE DIRECTORY admin_dat_dir
    AS '/flatfiles/data';

This directory can point to local file systems, NFS mounts, or even cloud object storage today (with the ORACLE_BIGDATA driver for S3, OCI, Azure). The DIRECTORIES don’t require moving data, though you could prepare those files via ETL pipelines or third-party tools, or they can be generated directly by applications.

We can now create an external table based on this directory, e.g., log files, bad data that we store externally, JSON files, and make data accessible inside the INFORMATION_SCHEMA and with plain SQL, as if it were internal.

Creating an external table:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

CREATE TABLE admin_ext_employees
                   (employee_id       NUMBER(4), 
                    first_name        VARCHAR2(20),
                    last_name         VARCHAR2(25), 
                    job_id            VARCHAR2(10),
                    manager_id        NUMBER(4),
                    hire_date         DATE,
                    salary            NUMBER(8,2),
                    commission_pct    NUMBER(2,2),
                    department_id     NUMBER(4),
                    email             VARCHAR2(25) 
                   ) 
     ORGANIZATION EXTERNAL 
     ( 
       TYPE ORACLE_LOADER 
       DEFAULT DIRECTORY admin_dat_dir  --notice this dir with above
       ACCESS PARAMETERS 
       ( 
         records delimited by newline 
         badfile admin_bad_dir:'empxt%a_%p.bad' 
         logfile admin_log_dir:'empxt%a_%p.log' 
         fields terminated by ',' 
         missing field values are null 
         ( employee_id, first_name, last_name, job_id, manager_id, 
           hire_date char date_format date mask "dd-mon-yyyy", 
           salary, commission_pct, department_id, email 
         ) 
       ) 
       LOCATION ('empxt1.dat', 'empxt2.dat') 
     ) 
     PARALLEL 
     REJECT LIMIT UNLIMITED; 

The first and most important choice is TYPE, which determines the access driver and what kind of files you can read: ORACLE_LOADER for plain text files like CSV or logs (read-only), ORACLE_DATAPUMP for Oracle binary dump files, ORACLE_BIGDATA for cloud object stores like S3 or OCI in formats like Parquet or Avro, and ORACLE_HIVE for Hadoop/Hive data. The DEFAULT DIRECTORY points to a server-side path alias, and LOCATION names the actual file(s), with wildcard support (*.dat) so you can load a whole batch at once.

The ACCESS PARAMETERS block is where you control parsing: row and field delimiters, null handling, custom date format masks, and where to write bad rows (badfile) and parse logs (logfile). On top of that, PARALLEL lets Oracle split file reading across multiple processes for large files, and REJECT LIMIT controls fault tolerance. Set it to UNLIMITED to skip bad rows silently, or 0 to fail immediately on the first error.

You see lots of built-in features that we can use compared to building a full-fledged data pipeline. Instead of exporting and importing CSVs from the source databases or developing a complex CDC pipeline that traditionally looked something like: source OLTP --> CSVs --> IDW (reports on yesterday) -> ingest into DWH for long-term analytics, we can just define a table based on external data and access it as part of our pipeline.

[!tip] The INFORMATION_SCHEMA analogy

You are probably familiar with the INFORMATION_SCHEMA of a database. It’s the internal data catalog that most databases provide and it contains a list of all tables and all metadata such as columns, data types, etc. The neat thing is that external tables will show up as internal tables once defined.

What’s the Modern Version of External Tables Today?

To preface: the previous Oracle example shows the CREATE EXTERNAL TABLE syntax, and a first-class DDL object in the data catalog. What follows in this chapter is the next evolution, where external tables are not necessarily created with DDL, but in another way, achieving the same outcome of querying data in place without loading it. Let’s see what these are.

Integrated into Warehouses

Most modern warehouses - Snowflake, Redshift Spectrum, BigQuery, Athena, Synapse - come with a simplified version of CREATE EXTERNAL TABLE. Compared to the Oracle example, the schema is usually inferred from the file format (especially Parquet), S3 or another object store is the default backing location, and the parsing ceremony disappears. The pseudo-code looks roughly like this across engines:

1
2
3
4
5
6

-- Pseudo-code: modern external table over Parquet on S3
CREATE EXTERNAL TABLE sales
WITH (
  LOCATION = 's3://my-bucket/sales/',
  FORMAT = 'PARQUET'
);

Object storage like S3, GCS, and Azure Blob has become the first-class citizen for external data. From here, the ecosystem layers on: dbt wraps this in YAML, DuckDB skips the DDL entirely in favor of schema-on-read, and open table formats add transactional guarantees on top.

External Tables with dbt?

On top of this base SQL form, dbt adds a YAML layer and can be used with its own package called dbt-external-tables. It’s one of the most-used dbt packages, though it seems less actively maintained now.

The external table is defined via YAML, and there are lots of options to set, with the most important being external and its location, but also defining columns in different ways such as inference or the meta tag:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

version: 2

sources:
  - name: snowplow
    tables:
      - name: event
        description: >
            This source table is actually a set of files in external storage.
            The dbt-external-tables package provides handy macros for getting
            those files queryable, just in time for modeling.
                            
        external:
          location:         # required: S3 file path, GCS file path, Snowflake stage, Synapse data source
          ...               # database-specific properties of external table
          partitions:       # optional
            - name: collector_date
              data_type: date
              ...           # database-specific properties

        # Specify ALL column names + datatypes.
        # Column order must match for CSVs, column names must match for other formats.
        # Some databases support schema inference.

        columns:
          - name: app_id
            data_type: varchar(255)
            description: "Application ID"
          - name: platform
            data_type: varchar(255)
            description: "Platform"
          ...

        # Use `meta` to pass custom column properties (e.g. alias, expression)
        columns:
          - name: raw_timestamp
            data_type: timestamp
            config:
              meta:
                alias: event_timestamp       # rename the column in the external table
                expression: TO_TIMESTAMP(...) # custom SQL expression instead of default value extraction

This is a nice improvement over the ODBC GUI interface. It’s not exactly an apples-to-apples comparison as dbt itself is not a database, but with its supported destinations such as Redshift (Spectrum), Snowflake, BigQuery, Spark, Synapse, and Azure SQL, you see that it will persist in these destinations, mostly data warehouses.

DuckDB with dbt

If you use dbt, you can also use DuckDB with dbt via dbt-duckdb, which is more up-to-date. But DuckDB is not an external table, right?

Yes, DuckDB doesn’t have CREATE EXTERNAL TABLE syntax yet, mostly because it is an in-memory database, but you can achieve the same functionality through other means. DuckDB can not only be used as a database but also as a zero-copy SQL connector (see all categories at 5 Key Categories). We can just point it to an external source, as shown above with dbt. The difference is that DuckDB is both a database and a compute engine, making ad-hoc reads possible directly without a DDL definition, similar to an external table with Oracle loaders. With dbt, we can nicely declare this in dbt configs.

With DuckDB, you can query “external data” extremely fast over HTTPS or locally in formats such as Parquet, CSV, and many more, so the need for formal external tables is reduced since DuckDB does schema on read.

If you want to define the database schema ahead of time, we’d use external tables to do that and effectively have schema on write (though we don’t write, just define the DDL table structure and data types), which is more of the classical ETL approach.

Here’s an example with external_location to read external data with dbt:

1
2
3
4
5
6

sources:
  - name: external_source
    config:
      external_location: "s3://my-bucket/my-sources/{name}.parquet"
    tables:
      - name: source1

Read more at Fully Local Data Transformation with dbt and DuckDB.

Other options are with database views that are supported in DuckDB with CREATE VIEW over read_parquet(). You can ship a .duckdb file to clients with pre-defined views over S3 data, so clients don’t need to know about the underlying data, Hive partitioning, or even glob patterns — very similar to what a formal CREATE EXTERNAL TABLE would do.

1
2

CREATE VIEW events AS
  SELECT * FROM read_parquet('s3://lake/events/*.parquet', hive_partitioning=true);

Or similarly use ATTACH to directly point to Postgres, MySQL, SQLite, S3, and others:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

-- Postgres (binary wire protocol, predicate + projection pushdown, read+write)
INSTALL postgres; LOAD postgres;
ATTACH 'dbname=postgres user=postgres host=127.0.0.1' AS pg (TYPE postgres);
ATTACH 'postgresql://user@host/db' AS pg (TYPE postgres, READ_ONLY);

-- MySQL (via MariaDB Connector/C; Postgres-style keyvalue string even for MySQL — easy trap)
INSTALL mysql; LOAD mysql;
ATTACH 'host=localhost user=root port=0 database=mysql' AS mdb (TYPE mysql);

-- SQLite (file opens directly; multi-reader single-writer by SQLite file locks)
INSTALL sqlite; LOAD sqlite;
ATTACH 'sakila.db' (TYPE sqlite);

-- Generic remote DuckDB file
ATTACH 's3://duckdb-blobs/databases/stations.duckdb' AS stations_db;

Open Table Formats and Lakehouse Architecture

That begs the question of whether Open Table Formats are the next evolution and modern way of external tables. These table formats allow almost any SQL compute engine to use them as external tables, and read, compute, and aggregate as a database would.

If we look at what table formats consist of, they’re built on object storage, with a file format like Parquet, and then we have a manifest file that contains a list of files that unifies multiple single files into a “single” table, looking from the outside.

So again, the manifest file is our pointer or fancier symlink, but it lives next to the data, unlike external tables. There’s much more going on in table formats, but if we have a data lake with open table format tables, we can see how we define tables in DDL and the pointers are to different files (Parquet, ORC, Avro), in most cases Parquet.

More broadly, we can say external tables decouple storage from compute. Open table formats decouple the table itself (schema, history, transactions, statistics) from any single engine.

Lakehouse and Connecting to DuckLake

One step further is obviously a lakehouse architecture, with the shift from format-agnostic file reading to governed, transactional, multi-engine open table formats.

If you extend the external table idea to a lakehouse architecture, these external tables with open table formats provide essentially what databases provide with ACID guarantees, time travel, schema evolution, partition evolution, and fine-grained access control, but for files.

But with the difference that data stays in open Parquet file format on customer-owned cloud storage. The external table, once a humble workaround for avoiding data loads, has become the architectural foundation of the data lakehouse if you like this analogy.

With DuckLake, we have the next evolution just around the corner, bringing back exactly that missing database, especially to handle all the metadata of such a lakehouse and all its files. This means having durable and consistent database storage for our manifest files.

Open Data Catalog to Complete the Picture: The ODBC Glue

With all these evolutions, we’ve come far. When adding an Open Data Catalog, we are exactly where we started: having an INFORMATION_SCHEMA, a dictionary with all our tables, in this case the open table format tables.

It’s the glue that ODBC provided when connecting a BI tool to the underlying database. Now you’d like to have an open data catalog that, in the best-case scenario, gives you all the tables and ways to connect.

But then again, the syntax of EXTERNAL TABLES still gets added, and ADBC and DuckDB are doing a great job of using external data without needing a data lake and its technology stack altogether. For example, DuckDB has support for ODBC, ADBC and even JDBC. That matters especially for 3rd-party tools: ADBC streams Apache Arrow end-to-end instead of serializing row-by-row, so BI tools and notebooks can pull millions of rows directly from external Parquet tables at speeds that previously required keeping data “hot” in a cloud data warehouse. 🙂

[!note] ADBC, what is that?
ODBC is 30+ years old, and we have a newer, faster version of it, called ADBC. It’s a faster way to connect to other databases with a columnar-oriented API instead of row-by-row serialization, heavily making use of Apache Arrow.

While ADBC is newer, it tries to support the same drivers as ODBC, but faster and easier to install. E.g., it has a handy dbc CLI to install it on almost any programming language, so no more manual and error-prone Windows GUI ODBC downloading of drivers and definitions needed, just one CLI command.

[!tip] Using MotherDuck
If you want a data warehouse that just works, integrates well with DuckDB, and has support for DuckLake, you can always use managed MotherDuck. You can build a classical data warehouse with plain SQL, you can read external data easily with DuckDB or dbt-duckdb, or integrate with DuckLake.

It works great with agents. Check out MotherDuck’s agent-skills for opinionated AI skills for building applications with MotherDuck. And visualize with Dives with one prompt.

Which Is Faster? A Quick Benchmark

To put numbers behind the hot/cold decision, I ran a simple benchmark on the TPC-H SF=1 lineitem table (6M rows, ~150 MB), stored four ways: inside a DuckDB file (internal), as raw Parquet, as an Iceberg table, and as a DuckLake table. Full code: bench2.py and metadata_bench.py.

Dashboard workload (hot path): 3 queries × 10 repeats:

Internal is fastest; external pays a 1.3×–1.7× tax. But for cold/archival queries (one-off, no warmup), all four backends answered in under 150 ms. The speed difference effectively vanishes for data you query once a week.

Storage cost is where external tables shine. Columnar Parquet is ~40% smaller than native DuckDB format. Ten TB of archive data costs roughly ~$125/month on S3 Infrequent Access or ~$10/month on Glacier Deep Archive, versus ~$230/month inside Snowflake on capacity pricing. This is the economic case external tables were invented for, and it still holds.

Metadata workload is where DuckLake stands out. Fifty single-row inserts showed DuckLake creating zero data files (rows inlined in the catalog) versus Iceberg’s 352 files (201 data + 151 metadata). That’s the “small file problem” made concrete: at one write per second, Iceberg creates ~86,400 files per day needing compaction. DuckLake creates zero until you checkpoint. DuckDB Labs’ own benchmarks report up to 926× faster queries on streaming workloads.

So Should You Use External Tables?

So after all this, should you use external tables today? After seeing how sticky they’ve been since Oracle 9i in 2001, how they keep getting re-added to newer tools (Snowflake in 2021, Databricks Unity Catalog, BigLake in 2022), and how their core benefit is. Accessing data where it lives without moving it, via a simple DDL statement, has only grown more valuable as formats have evolved from CSV to Parquet, JSON, Avro, and now open table formats. I’d say yes. But choose wisely based on your data’s temperature: use internal storage for hot data, such as dashboards and frequently used queries.

Use external tables for cold data, archival workloads, and ad-hoc exploration, where that gap vanishes, and storage costs plummet (up to 20× cheaper on Glacier Deep Archive vs. warehouse-managed storage). And if you already use dbt, DuckDB, or a lakehouse stack, the modern versions are right there. Where they’re the wrong choice is the inverse: transactional workloads, queries that need sub-second latency on every run, or data so small that the operational overhead of an external stage outweighs the benefit of not loading it.

The evolution is worth naming explicitly: “read CSVs on disk” → “read Parquet on HDFS” → “read Parquet on S3 via a metastore” → “read Iceberg/Delta tables with ACID on S3” → “the Iceberg table is the warehouse table”. Each step kept the core idea (data stays where it lives, metadata describes it, SQL queries it) and added database semantics back in. With open data catalogs, the warehouse becomes a stateless rental over a bucket you own, and external tables are increasingly managed. DuckLake demonstrates this best: when the catalog has SQL-DB-like guarantees, the distinction between “external” and “internal” dissolves. The metadata benchmark made this concrete by reading a single indexed row rather than walking a manifest tree.

The database semantics are returning with DuckLake, managed Iceberg, and predictive optimization, all of which reintroduce RDBMS-style guarantees to the lake. The cycle from “external table for cheap storage” to “external table as a full ACID database on S3” took 25 years, completing the journey back to database principles while maintaining the separation of storage and compute. You can say the modern external table isn’t external anymore. DuckDB reads them directly, and DuckLake handles the metadata that multifile lakehouse architectures would otherwise drown in. The lesson from history is that whenever someone tries to replace it, the pattern is that reading data in place always beats moving it. And the Lindy Effect suggests that if external tables have lasted 25 years and get re-added, they’ll persist another 25. They’re probably not going anywhere. 🙂

Full article published at MotherDuck.com - written as part of my services

This article looks at how BI evolved, how dashboards are actually used today, and what survives when AI enters the picture — starting with the foundation that was never really about dashboards in the first place, and ending with the problem nobody in the AI hype cycle wants to talk about: who maintains it all.

The Verdict of People in the Field: BI is Dead (Again)

We’ve heard it all. Business intelligence (BI), and especially dashboards, are dead. But every time, only to rediscover its power and resurrection whenever we need grounded data analysis in any enterprise and startup space. The same way Excel never dies, which arguably is still the most used BI tool.

If we look at what others from the data world say, it does sound similar. Hex says that dashboards were never the destination and specifies:

Static reporting surfaces were always a workaround for messy data and limited tooling — agentic analytics finally closes the gap between visibility and genuine insight.

In other words, dashboards were a workaround created because data was messy, tooling was restrictive, and asking open-ended questions of the warehouse wasn’t possible without coding. Hex continues that dashboards create more questions than they answer.

But they also say: respect dashboards, but stop treating them as the goal:

Dashboards still matter. They’re excellent for reporting KPIs, surfacing operational signals, and aligning leaders around shared metrics. They allow data teams to be creative in how they display these metrics and will remain a primary surface for need-to-know numbers.

Dashboards don’t reason or explain why something happens, as the Hex article says. Hex argues that reliable, timely, context-aware answers are the destination.

Mike, CEO of Rill, says:

Agents are blind—they can’t see dashboards. But they do need access to the primitives behind them. AI, and agentic working won’t kill BI, but it will make dimensional modeling, metrics, OLAP cubes, query performance, and governance more important than pretty charts. Agents will reveal what many of us have known for a long time: BI was never about dashboards.

Benn Stancil was already saying in 2021 that BI is dead, where he drew parallels to the Salesforce “End of Software” declaration in 2000. This is very interesting as it’s almost the same statement today with AI, no more software developers needed. Benn argues that the original BI stack was one tool across the stack, but that by 2021, tools were unbundling into dedicated and specialized tools for each layer.

The proposed future of BI should focus only on data consumption for humans, integrating both self-serve applications and deep ad-hoc analysis. This new BI would be “legless” (the opposite of headless, he argued back then), relying on global governance layers rather than proprietary semantic layers, and fostering cross-functional collaboration.

The aim is a universal tool for all data consumption, moving beyond its current diminished state of “visualization and reporting”. Again, very similar to today’s discussion, where everything is about semantics and context. Benn concluded that “BI is dead, long live BI” 🙂.

Take any modern data and analytics discipline, and you’ll probably find it has its roots in the work that has been historically carried out by Business Intelligence developers, the OG jack-of-all-trades of the data industry.

[!note] Find more Opinions and Articles

Other opinions on the world wide web, whether BI is dead. Long live business intelligence., or a discussion on Reddit about Will Business Intelligence skills (BI) be irrelevant in like 3-4 years? : r/analytics or AI kill BI?.

BI Was Never Just About Dashboards

With respect to Benn’s article in 2021, have we come full circle five years later, with the end of software engineering again, and everyone demanding semantics and metrics layers?

At least on the surface, it seems we are at the same point, but today we’re going back to one “Original BI” stack as drawn in the image in the article, to a fully encompassed data platform. Maybe it does not need to be one single platform, but at least to the user it needs to be a single chat or AI interface, that goes end-to-end through all the layers of discovery, visualization, transformation, storage, and ingestion—or in other words, the full data engineering lifecycle.

So what about dashboards then? Many declare the death of many things, and dashboards are a popular target too. That’s even more true when AI, with its generative capabilities, can just one-shot your whole dashboard and create a full-blown web app or custom HTML page. And dashboards might die, as many don’t actually want the dashboard itself, but the extracts from your large SAP, linked to the right customers from the CRM, enhancing the decision-making process even more.

They want the insights from the combination of all source data and business insights that your company has over its competitor, for example. It’s the primitives behind the dashboards that matter more.

When You Still Need a Dashboard (AI Chat is Not Enough)

Even though a chat interface or an agent can provide you with dashboard information tailored to your question and in an explanatory written form, there’s still a need for dashboards in certain situations.

The obvious one is the well-crafted operational dashboard, where you can see your whole company performing in a split second by looking at a single, highly dense dashboard with individual charts and visualizations tailored to convey information about each sub-area in the best possible way. It’s the same way a map is still needed in self-driving cars: for quick verification, to get an overview, or in case the car gets lost.

The other obvious benefit of the operational dashboard, or any general dashboard, is that people can agree on numbers, as they’re looking at the same agreed-upon dashboard. Hence, the calculations are the same compared to individual Excel sheets with different calculations.

The easy creation of multiple dashboards on the fly or chatting with AI to get insights resembles the old way of using local Excel files. Everyone is doing their own thing, with no alignment, governance, or broader verification.

Maps are a different type of dashboard that can’t be replaced. Geospatial data shown on a map still beats text. Or a bar chart where you immediately scan the different proportions of stock in inventory across certain regions, compared to first having to analyze all numbers in a text-only chat reply.

Dashboards as a Sanity Check

Maybe some even less obvious reasons: serendipitous discovery of anomalies and outliers that accidentally pop up in dashboards are harder to see in chats, but easy to spot visually. The same goes for ad-hoc BI when drilling down into more grain and dimensions for self-service users. A pivot table is a REPL for BI; that’s not possible with chats.

Dashboards are also a lifeline to quickly check if the AI is hallucinating. What about determinism? Chat responses are not deterministic, and you might get different responses, hopefully with the same correct answer, but visualized differently because the model made different decisions the second time, or for a different user. AI agents are non-deterministic statistical models and probably always will be, so we need to bring more context and definitions to make outputs more consistent. One way is with a spec-driven development (SDD) first approach that helps define it more accurately each time.

The same applies to stewardship, reviewing and checking outcomes. As an example consider self-driving cars such as Waymos: you don’t need a map, but if it’s wrong or stuck, the first thing you’d reach for is one.

But what does that mean for the future of BI, and what keeps BI afloat?

The Primitives of a Dashboard (and BI)

BI was and is never (only) about dashboards. BI is used for a combination of reasons. One of the most important is the metrics themselves, the business KPIs: defining your profit, defining what has to be deducted, how much taxes and salary payments are. All of it is ingrained in a single metric, or better, all of it including the hierarchy, the full tree of metrics, as it’s never just one metric.

But the metrics tree with all its hierarchy is not all. We need speed to crunch the data extracted from the source SAP and CRM, cleansed and joined, and aggregated to the exact grain of the metric. No one has a single place to view their data end-to-end, which makes BI work so needed, integrating to either combine all sources into one API or database/warehouse for the business (or now AI chat) to pull from. Again, queries over CSV are easy. Fast queries over 1 TB of Parquet are hard.

This is where we need optimal data modeling and compute power to do it in seconds, best case sub-second. And when we have this, we need a good data architecture with the right tool to compress and compute that data. That’s where we need some kind of OLAP or analytical database, optimized for analytical queries and doing aggregation on the fly based on different dimensions and grains.

All of these primitives are still needed, maybe even more so in times of generative AI, even if dashboards were to go away.

Lastly, all of this is under the umbrella of context and semantics. It’s encoding the business-related processes into data-like artefacts to make reusable governed definitions of metrics like revenue, MAU (Monthly Active Users), ROAS (Return on Ad Spend), etc. And the right medium for that is metrics and a so-called metrics layer (or semantic layer).

Everyone Wants to Build Dashboards. Nobody Wants to Maintain Them.

The elephant in the room that nobody talks about is that everyone wants to build. Nobody wants to maintain. This is the current AI phase we’re in. Even more, nobody even wants to review tons of AI-generated code.

According to the story in the book Maintenance of Everything by Stewart Brand, what makes people do maintenance is when they find joy in it and care for it.

This is greatly illustrated in the opening with the 1968 Golden Globe solo sailboat race, a dramatic contest of maintenance styles under life-or-death conditions, contrasting Robin Knox-Johnston’s meticulous upkeep with Donald Crowhurst’s neglect. Knox-Johnston, who truly enjoyed his boat Suhaili, reported mid-race that despite the brutal Southern Ocean ordeal, he was “thoroughly enjoying himself”. Decades later, he personally restored her, replacing every one of her 1,400 fastenings.

Stewart Brand also argues that you need to care about your product, and that’s the key to good maintenance — a lesson drawn from Zen and the Art of Motorcycle Maintenance by Robert Pirsig (1974), a philosophical classic celebrated as bending the culture of the day toward honoring maintenance.

And one way to ease maintenance is to design maintenance-friendly. Like the Rolls-Royce Silver Ghost, which was engineered from the ground up for reliability and ease of upkeep.

So how do we apply these principles to BI? We need to create dashboards that are easy to maintain and have clear ownership, someone who cares. Otherwise, they lose long-term value, and we invest precious time in something short-term.

[!quote] Quote by Mehdi
You just burned $1k in tokens to rebuild the SaaS you refused to pay $29/month for.
Great for the ego. Now maintain it for 3 years to break even.
Yup, maintenance is a luxury these days.

Generating 100s of cool but unused dashboards with AI is clearly working in the opposite direction. We can save a couple of bucks while no longer needing to maintain each custom-created dashboard. Indeed, maintenance is a luxury these days.

Self-Maintaining Agents?

The question is, do we need agents for the maintenance then? So that we can create new, innovative solutions, do the creative things, and leave the hard parts of BI to us humans such as getting requirements and verifying with the business. And we create agents to do the maintenance? What does maintenance even mean? What does changing the oil and checking the brakes mean for BI, or data pipelines?

It’s not troubleshooting in case of an error, that’s what agents already do and help a lot with, ideally fixing the errors themselves, in a self-healing process. But maintenance is different. Keeping up with the updates of your software, security, the right integration into your data platform with upgraded, better-performing glue code, avoiding code from becoming legacy code.

All of it means maintenance, and outweighs the work of just creating the data pipeline or BI dashboard by far, probably somewhere on the order of 8:2.

As Mike puts it well when we were discussing:

Few can build a maintainable, scalable data infrastructure for surfacing trusted metrics. E.g. a digital platform like Coinbase isn’t going to YOLO its internal reporting over billions of transactions. Even Claude has a usage-based billing portal, consumption metrics need to be precise, deterministic, and fast.

The Basecamp owner similarly said: AI is pushing back too little. Maybe it will be solved in the future with better models, but we need to live in the present, and in that present:

Agents don’t finish beautiful, ergonomic, desirable software. They just don’t. That human finishing at the end is not just necessary, it’s essential.

So the future is soon, but not yet. Back to dashboards and their maintenance: the hard part is not generating visualizations, but having metrics and a strong BI backend. Almost a unified data interface that has an agent with access to source, ETL, and dashboard.

BI-as-Code: One Solution to Maintenance?

Is the solution to maintenance-friendly design maybe BI-as-Code?

BI-as-Code comes into play because declarative configs can be versioned and maintained, thereby avoiding the limitations of BI-as-clicks. Sure, it will not solve all problems, but having that descriptive state of A. your data infrastructure and B. your data pipelines and BI dashboards helps tremendously. In the event of an error or incorrect state, we can just roll back to the last versioned dashboard or infrastructure.

The only thing hard to make reversible, unless you use some kind of Git for Data workflow with LakeFS, Nessie, or others, or just use Open Table Formats with the Time Travel function, is the actual data.

BI-as-Code isn’t the whole answer, but it’s the right direction: making dashboards owned, versioned, and recoverable. Code can build the right level of abstractions for ETL, metrics queries (metrics SQL), and visualizations, where raw Python for ETL or D3 is too verbose and too brittle.

With agents, these abstractions come in handy once more, as agents work best with a clear interface like a CLI or API, where the abstraction helps build just that, and tune things themselves through MCP or direct access to the declarative configurations. Much of what GenBI is all about. The question of what comes next: can agents take over the analyst role entirely, or how do we marry the two?

Building BI for Agents, Not Humans

BI-as-Code allows agents to drive BI, or as Mike said: “AI drives compression of the data stack”. Meaning observability, cybersecurity, product analytics, and BI are converging. A CEO recently asked him why he couldn’t “kill Tableau, Looker, DataDog, Grafana, and QuickSight” in favor of a single system. In my opinion, it doesn’t need to be a single tool, but it should feel like a single interface.

Most common today: a chat prompt that autonomously spawns ingestion, transforms data into marts, and surfaces a dashboard or web app, running end-to-end analytics without the user ever thinking about the layers underneath.

But only the speed of faster building with AI won’t get us there. Amdahl’s Law still applies, as Jeff Dean rightly said in his talk with NVIDIA’s Bill Dally:

An AI agent can run 50x faster, but if the tools it depends on were designed for human speed — slow query APIs, brittle CLIs, unversioned metrics — the overall gain collapses to 2-3x.

That’s why the primitives behind BI get more important as agents get faster, not less. OLAP needs to be faster, metrics need a reliable API, ETL needs to be composable. The bottleneck shifts from the model to the infrastructure it runs on.

And when agents do take over the analyst workflow, spawning parallel queries, discarding dead ends, surfacing the interesting slices, they’ll expose something BI practitioners have known for years: the hard part was never the visualization.

It was always the semantics beneath — the governed metrics, the trusted definitions, and configs that were verified by an actual human being. Agents will just make that gap impossible to ignore.

BI primitives are Infrastructure for AI

Wrapping up, BI was never about dashboards. It was about making sense of a company’s data, connecting the source data into something a human can understand, efficiently reusing existing metrics, and governing definitions.

The dashboard was just the visible surface. What survives the hype cycle, from the unbundling of the modern data stack to the rise of AI agents, are the primitives: metrics, semantics, ownership, trust.

The AI era doesn’t kill that need. Agents hallucinate without a strong foundational semantic layer or verified human constraints. Non-deterministic chat interfaces collapse without business-wide, agreed-upon definitions. The maintenance problem doesn’t disappear either when you generate faster. It compounds the problems and bottlenecks for senior engineers at a company.

BI-as-Code, versioned dashboards, and a governed interface aren’t opposite to the AI future, but a necessary foundation that makes working with it easier, not only for AI systems but also for humans in the loop.

–

If you enjoyed this, there’s further related reads that might be interesting to you:

• Data Modeling for the Agentic Era: Semantics, Speed, and Stewardship
• Has Self-Serve BI Finally Arrived Thanks to AI?
• BI-as-Code and the New Era of GenBI

Full article published at Rilldata.com - written as part of my services

You might use AI agents all day long, parallelizing them with AI orchestrators like Agent Teams, Gastown, tmux, git worktree, and AI-based IDEs, but in the end, you just coordinated an AI. You still have to learn what it created, understand it, check for hallucinations, and verify that it built the right thing. We’ve all become senior reviewers, more exhausted than before, with less of the work that made this fun in the first place. Meanwhile, we are also more distracted than ever. No time to think, with Copilot, Grammarly, or something else constantly asking and suggesting.

This series interviews real practitioners to give you the best tips on how they use AI in their data work today, extracting as many patterns behind them as possible. The article is structured in four parts: (1) how Mark is using AI, (2) what he has learned working with it, (3) what he is specifically using it for, and (4) what he thinks about AI in general and the future.

Introducing the Guest: #1 Mark Freeman

The start of this series is none other than Mark Freeman. He is currently the Head of DevRel, Employee 1 and GTM at Gable. Mark has gone through three career roles as clinical researcher, data scientist, and data engineer, which is helping him greatly in his current position to navigate the unknown of generative AI. We’ll go more into it later.

Mark has also co-authored a book with O’Reilly about Data Contracts (with Chad Sanderson and B.E. Schmidt), and is helping build Gable with the best possible data flows and data quality for enterprises.

[!abstract] TLDR

To set the stage, in this interview we talk about how to use Spec-Driven Development workflow with Claude Code and agent teams to produce high-quality, reproducible outcomes. We cover Mark’s use of ExcaliDraw diagrams and JSON schemas to define requirements upfront, how he parallelizes agents with tmux to compare outputs, why AI benefits senior engineers more than juniors, and where he sees data engineering heading.

How Mark’s Using AI

Let’s start with the general setup Mark uses when working with AI, and how he uses generative AI.

How Mark Changed His AI Workflows

I asked him: “Since you’re building a company in the data contract and quality space and have written a book, how has working with AI changed how you use AI at work?”

Mark has been in the data space since 2018 as a clinical research analyst and a data scientist since 2019. In 2022 he shifted over to data engineering, and in 2023 joined Gable to solve the problem of applying data contracts. He was very early in NLP with the major ML project he worked on back in 2021.

He remembers the early days in 2023 when ChatGPT hallucinated and when he used generative AI for the first time. Very much as a chat window co-coding companion, asking them architecture questions and general questions about the code at hand. Fast forward to 2024 and 2025, generating more code, but not full programs and projects, but by function - trying to narrow down the scope.

And then in late 2025, Claude Code came around, and changed the game with better models that could autonomously solve problems for a longer period. And at the same time, everyone provided more CLIs to empower the CLI-first workflow of Claude. Mark started building by giving it instructions, pointers to docs, schema, etc., and letting it independently build data-related work and go fully agentic.

Mark’s Spec Driven Workflow

Mark has figured out a very well-working approach that helps him create reproducible outcomes. Not focusing on solutions, but on how the tool works as he relentlessly specs and defines what he wants with the Spec Driven Development (SDD) approach, inspired by Esco Obong and how he used it at Airbnb. He uses the GitHub-provided spec-kit, which is a toolkit to help you get started with Spec-Driven Development.

I hadn’t heard of it, and when checking it out, it’s super well documented and integrates 1:1 into Claude Code (and many other AI agents), meaning you can use slash commands within Claude and define specs with the help of an existing git repo including docs and code such as:

• /speckit.plan: Execute the implementation planning workflow using the plan template to generate design artifacts.
• /speckit.tasks: Generate an actionable, dependency-ordered tasks.md for the feature based on available design artifacts.
• /speckit.specify: Create or update the feature specification from a natural language feature description.
• /speckit.analyze: Perform a non-destructive cross-artifact consistency and quality analysis across spec.md, plan.md, and tasks.md after task generation.
• /speckit.clarify: Identify underspecified areas in the current feature spec by asking up to 5 highly targeted clarification questions and encoding answers back into the spec.
• /speckit.checklist: Generate a custom checklist for the current feature based on user requirements.

You can define these on a per-project basis, or have some of them defined as a general spec in your ~/.claude/ folder. The outcomes are Markdown files that hold dedicated specifications, based on your goals that can then be further edited and updated based on your iterations.

Working Product-Focused

This helps Mark to focus on product scenarios and predictable outcomes instead of vibe coding every piece from describing his principles from scratch, he continues.

He goes from ideation through specs to dedicated tasks. He likes to always start with an ExcaliDraw diagram, defining more of the flow diagram, rather than architecture or other overviews. For data schema and interface definitions, he defines data structure next to the relevant flow diagram, as ExcaliDraw is JSON, these can be easily integrated. Schema definitions describe accurately what’s needed based on stakeholder discussions and his needed requirements.

He then passes that diagram back to Claude Code and iterates on the data model and his key assumptions. Mark takes a lot of time in this process. He will spend hours, days or even weeks in this stage, updating and refining these specs, specifically giving clear and exact information about data schema, tools to use, and architectural choices that he knows as a senior engineer that he wants and needs to have.

This is also where years of experience make the difference.

Using TypeScript for Data Schema Enforcement

An interesting discovery Mark made is that he started using a programming language new to him, TypeScript. Similar to Wes McKinney’s From Human Ergonomics to Agent Ergonomics, where he states that “Python Was Built for Humans, Not Agents” and argues that he is using GoLang and Rust for agent work, as it’s a better language for agents with minimal dependencies and shorter/better types.

Mark ended up using lots of TypeScript, not because he was familiar with the language, but because it’s mostly what his work and that of a data engineer requires: defining data types. Enforcing them, quickly verifying across the data pipeline that we don’t get an error before pipeline runtime. Saving a lot of time and upping the quality.

What Mark Has Learned Working with AI

Over the years, Mark has changed his workflow. In this part, he shows how he uses agentic agents with tmux and how he reviews and checks the outcome.

Agent Parallelization and Executing Them: Teams and Tmux

After all the specs and focusing on them once, he uses agents to implement the specs and Claude uses the feature called Agent Teams (which can be activated in Claude settings.json with CLAUDE_CODE_EXPERIMENTAL_AGENT_TEAMS).

The cool thing about agent teams is that they let you coordinate multiple Claude Code instances working together. One session acts as the team lead, coordinating work, assigning tasks, and synthesizing results. Teammates work independently, each in its own context window, and communicate directly with each other.

Mark spawns multiple agents using iTerm2 and tmux, which I heavily recommend for agent work (also check Zellij for an easier start), and the agent teams feature will automatically open the additional terminals in separate panes:

It shows Claude self-orchestrating his own team. Think of it as similar to Gastown, Agor, and other AI orchestrators, but integrated into Claude.

Mark’s workflow with agent teams is deliberately outcome-focused rather than code-focused. Once the agents complete their run, he checks the result against the original specs and JSON schemas, not the code itself. The only thing that matters is whether the outcome does what was defined.

Is Reviewing Code Still Needed?

The tough question was whether Mark still reviews code, especially when Claude can generate more of it in a minute than we can ever review. Mark said: “Not locally or on unimportant projects where I’m exploring the limits and potential traps of these powerful tools.”

But for production pipelines or when customers asked him specifically for his opinion, he said:

Along with the wider industry, we are figuring out how to use AI safely at scale.

Also at work when they have mission-critical services such as in a bank, you can’t just vibe code something. It comes down to use-cases, he said.

Besides use cases, he tried different ways of reviewing. First he tried a sophisticated process where the above agents would create PRs and he would then comment on these with improvements and changes. The agents would then read them and integrate the given feedback and continue the process. But even that workflow made him too much of a bottleneck. It wasn’t scalable enough.

Mark searched for other ways to work with it.

Outcome-Driven Reviews: And Starting from Scratch Again

What he does now is assess outcomes instead. After all the rigorous time in speccing, he tests the result by running the pipeline, creating tests, or checking the code manually the old-fashioned way.

The key mindset shift here is that the first build is deliberately treated as throwaway. It’s requirements exploring via building. You implement the spec once, learn what you got wrong, and expect to discard it.

That’s why he tests the outcome. And once tested, he might have gotten new learnings that he could have only gotten through implementing or with actual tests. That’s when he will feed these learnings back to the specs and update initial requirements, and start all over again, from scratch, letting the agent create a new outcome based on the updated specs. The cycle is: spec → build → assess → improve spec/assumptions → repeat.

This way, he has an approach with a very deep and exact iteration, almost deterministic, where he can re-run the agents with updated feedback and requirements, and get the same or similar outcome with the added updates, because of the spec-driven approach and the structured approach that spec-kit delivers, and the dedicated way he defines his requirements, which won’t just be hallucinated as different inputs, end-to-end.

Though this can always happen, this approach served him very well, with a high-quality output he can trust, and a qualitative way to approach a complex problem with the help of agents.

If the outcome meets the quality he expected and it does what he wants, he goes to internal stakeholders to get feedback from them. And then the same process again, updating specs, fixing requirements errors or possible wrong assumptions, and off the agents go again.

Tests and Quality Gates

Tests and QA he writes manually. This is another way to make sure the outcome meets his expectations. Most important is the value, he says:

Value first, then outcome and then worry about other things

If it’s not turning out to be valuable to the stakeholders, he wants to avoid spending more time. That’s why the agent iterations and building something “quickly”, with rigorous specs and definitions in place, worked well for him so far.

Senior vs. Junior: Working with AI

We move on to an interesting discussion of whether AI helps senior engineers or juniors more. Mark says (he also wrote about it) that AI helps more senior engineers, as seniors “understand the trade-offs of tech debt”.

He says further that in AI iterations, we move much faster, generating legacy code and architecture constructs in days and weeks, instead of years. If Mark iterates with the spec-driven design explained above, there are multiple different architectures generated, some of which might have been bad from the very beginning.

As a senior, he thinks that we can give the right guidance from the very beginning and exclude bad outcomes and early “legacy code”. No doubt, there will be code and architecture to be adapted, too, but if you lack experience, you basically have no chance of knowing.

Framework and Architectures Are for the Experienced

Mark mentions that at Gable, he is building something from scratch. Let’s say we are at iteration v4: deep technical architectures are coming up, to choose an Apache Kafka infrastructure, define your schema in JSON or Avro, or use Parquet.

These decisions can only be made with experience. Sure, agents will give you a good middle ground, and with research they will potentially choose the right solution for the current problem. But how do you know what’s the best solution for your given business problem? If you have built multiple data platforms and have seen many companies, you just know some of these things or developed an intuition for what’s needed.

In combination with the agents, it’s just a much better tool for seniors than for juniors who need to more or less blindly trust the assessments the agents made. The quality of outcome depends on frameworks and architectural choices, accumulating legacy code early if a big architectural component is chosen wrong.

In a related but further way, the knowledge is like a linter in an editor that knows things ahead of runtime. It can detect wrong choices directly.

What Mark is Using AI for

Besides the already discussed use cases of general workflow and reviewing outcomes, I asked him about how he uses AI at work, working with data contracts and the non-deterministic outcome of AI, for example.

Integrating AI into Data Contracts

As an author of a book on data contracts, and working in the business, one of Mark’s priorities is to somewhat safely use AI agents to either verify contracts or help define them, if in any way possible.

As data contracts are written definitions between two parties, mostly written in YAML or JSON, it’s a good medium to iterate on, where agents, humans, and all stakeholders can work on specs that can be versioned. Mark says his focus is on evals, specifically for assessing how well an agent completes a specific task, built around Gable’s products or internal workflows.

The main goal of evals is to more confidently know that what AI shipped is any good. Similar to stewardship in Master Data Management (MDM), where humans in the process verified if the data quality was met, with AI generation we need a similar process at a faster pace.

That’s also where he draws on his clinical background with an outcome-driven approach, comparing 200 observations from end-to-end coding agent simulations and assessing results against defined criteria. At Gable, they create a Code Graph that helps them get a skeleton view of the full data flow in code, without running any code. Connections, context, and business operations are expressed as code to be verified.

His hypothesis is that with agents at scale, we can gather datasets of behaviors such as logs of data pipelines, network logs, and other information such as agent trajectories and check based on them whether the data pipeline is compliant, like A/B testing AI Agents with a Bayesian Model. This has yet to be proven, but the hypothesis is strong.

Deterministic and Non-deterministic Work in Data Engineering

When asked about his thoughts on functional data engineering where usually jobs are reproducible and restartable with new logic/source data, and how he sees the determinism with AI work (which has a different outcome every time), he said something interesting.

He said non-determinism is a benefit. That’s why the setup is specs written in markdown, combined with configs and JSON that are specific, providing precision and accuracy. If anything goes wrong or not according to plan in the generation phase, he can just change the specs and achieve this determinism by spec-driven development.

But there are still some disadvantages from running non-deterministically, that’s why he still does tests and comparisons manually, and checks visually whether everything works when running the pipeline.

What Mark Thinks about AI

When talking about the future, learning with AI or where it leads, or also when not to use AI, is what we discuss here.

When not to Use AI

Starting with when he is not using AI, and when it’s potentially cheaper or better to do it manually, his answer was:

Requirements finding in an important project, again depends on use cases. For small non-personal projects, not a problem. But requirements need to be defined by stakeholders and come from a real problem

Also, Mark mentioned key decisions for infrastructure code that needs to be stable and reliable. Or if used, he will spend much more time validating that LLM suggestions are correct.

For content online, he noticed that the writing always comes off differently than he would have phrased it. He might give it his insights to check or get feedback, but not the actual writing part.

How Do You See Learning with AI?

There’s also the danger of not learning new things, and getting overwhelmed with constant stimulation, potentially getting slightly addicted. I asked Mark if he sees a problem in using agents and LLMs that would prevent us from learning new things as we are just cruising on auto-pilot.

Yes, he agreed. He calls it: “Claude code slot machine”, or “Lab rat”. “Getting your dopamine hit beyond usefulness” is how he would phrase it. He also thinks that this addictive behavior doesn’t exist randomly. He thinks it is intended for us, the users, to use and spend more tokens (ergo money for them).

[!note] Pseudo Work

Shipping lots of code with AI can feel like deep work, but if you’re not learning in the process, it’s pseudo work. Problem-solving skills in adults are already declining, and even studies showing short-term learning gains with AI find that beyond 8 weeks, the effect reverses as over-reliance sets in.

The Future of Cloud vs. Local Model

My closing question was where things are heading, and whether self-healing data pipelines would be a thing. When some say that “Unironically, Rick Rubin is the future of work” (where AI replaced a team of analysts, a strategist, a designer, a project manager, and a few weeks of work in minutes), the same goes for data analytics and data engineering.

Mark explains that when he was a data scientist, getting a nice histogram in Matplotlib or Seaborn took hours. Today he gets that for free, as an afterthought. He has built applications that pull leads from Hubspot, extend and aggregate data through RAG using APIs and pipeline logs, and for a board meeting just generate a static HTML page (with an export to CSV 😉). A custom-made visualization at your fingertips. That’s the future, he says. Because below the visualization, there’s a semantic model as the base. No one wants to open one more app, so based on well-defined semantics, AI can one-shot the visualization and integrate into existing workflows.

On the local model side, another future he sees (and is exploring himself) involves models running on a dedicated machine while he’s away. He said the future is not about how powerful the models are, but how many iterations your spec has gone through. You run them until they are correct. You can also use RAG techniques to augment the model with your own notes and skills, one local model custom-made for you:

You can’t compete on compute, but you can use the factor of time, iterating multiple versions for a specified problem, and choosing the best one. Exactly what clinical research is doing and what he learned in his early career comparing studies.

An interesting bleeding-edge area is running agents optimized for concurrency, chunking tasks and parallelizing them with smaller compute resources instead of one big model. Abi Aryan is doing GPU research in exactly that field, and Mark recommends starting with this post. While companies are paying 10x or more for cloud compute, local models with lots of iterations are increasingly feasible, and the economics are starting to make a strong case for them.

Next Interview

I hope you enjoyed this interview with Mark. Huge thanks to Mark for taking the time to speak with me and for sharing his experience with all of us. Follow him on LinkedIn and his Course on data quality and check out his book, its repo, and much more.

There are three more interviews already lined up with great guests, so please share feedback, questions you might want to ask or just your experience on how to work with AI in the data space. We’re all in this together, figuring it all out. The more we can learn from each other, what’s important, and maybe also what’s not, the better.

So stay tuned for the next interview.

Full article published at MotherDuck.com - written as part of my services

When Rill bet on YAML dashboards and CLI-first workflows in 2022, they weren’t thinking about AI agents. Neither was MotherDuck when they built serverless DuckDB around the thesis that most data fits on a laptop. But it turns out, what is developer-friendly is also agent-friendly, with the needs of readable code, fast engines, and deterministic semantics.

Times are shifting rapidly toward CLI-first development. You know that’s true when even email and calendar get their own Google CLIs. So why not have CLIs for your business metrics too?

This is what Rill and MotherDuck provide, including excellent developer workflows with a local-and-CLI-first approach, focusing on a developer-friendly interface and empowering users. Both work great on local laptops but can easily scale to the cloud, backed by a serverless data warehouse..

The convergence of embedded analytics engines (DuckDB/MotherDuck), declarative BI-as-code (Rill), and AI agent protocols (MCP) is creating a new architecture for business intelligence, one where dashboards become code, code becomes agent-readable, and analysts shift from clicking to prompting. And with 75% of cloud data warehouse queries scanning less than 1 GB¹, this opportunity is great for agentic BI. In this article, we look at how we build agentic-friendly and local-first analytic stacks with MotherDuck, Rill, and agents.

[!note] End to end examples later in the article

Later in the article we go through three different examples of how this can work, including GitHub repos and code examples, if that is something that’s of interest to you.

Why These Two Tools?

Let’s start with why do we use MotherDuck and Rill for agentic-first data tasks? As Ghanshyam Chodavadiya from SWYM says:

[!quote] Quote on why SWYM use Rill with MotherDuck for their AI-native media decision platform:

[..] Rill lets us encode business context directly into our BI layer. Combined with MotherDuck and the Rill MCP client, it gives us flexible data control while powering automatically generated client dashboards and AI-driven insights.

Both MotherDuck and Rill use a sophisticated architecture that focuses on developer workflows and scales from local development with declarative configuration to cloud (e.g. with rill deploy, or md: instead of md) or even embeds into your data CI/CD or agents pipeline, very easily. All of these reasons make them suitable for modern data requirements, where we need to iterate quickly but still have a strong foundation.

Local-first Approach with DuckDB

Both tools start from a local-first approach with DuckDB as the foundation.

For example, the Local-First principle, that was tackled by Ink & Switch and its community compares different strengths of local workflows meant for files, but also applies to data workloads. Even more so with AI agents, which can read the context from these projects and enhance easily with the use of strong CLIs that are available on the command line.

Or reading the Small Data Manifesto by MotherDuck that says “Think small, develop locally, ship joyfully”. If you enjoy some of these principles, this data stack with DuckDB/MotherDuck as the warehouse or real-time analytics storage and Rill as an interactive, fast, and beautiful BI tool will suit you well.

MotherDuck works seamlessly through the DuckDB CLI, whether it is to connect through their serverless database in the cloud (connect with duckdb ':md') or to open a fully fledged notebook environment locally with duckdb --ui (try it).

With Rill’s YAML-based dashboard and metrics layer, and a powerful CLI, you can transform any of your data into a blazingly fast dashboard locally (run rill start), or from anywhere with the data on MotherDuck. Let’s explore both in more detail, and show how users use it, and provide an example for you to get your hands on.

What Is MotherDuck?

Before we go into the hands-on examples, let’s answer the question of what MotherDuck and Rill are. And what’s the difference from DuckDB and what do they bring to the table?

In its essence, it’s a DuckDB-powered cloud data warehouse that scales to terabytes with ease. Just as Turso hosts SQLite, MotherDuck hosts DuckDB in the cloud, serverless for you. MotherDuck has great relation to DuckDB Labs, the company behind DuckDB and the DuckDB Foundation.

MotherDuck integrates well with DuckDB, but you can also just run DuckDB locally without it and manage your server yourself, open some ports, make it scale automatically if more queries come. But you’d need to create an orchestration that scales out, handles OOM, servers, etc. So instead, MotherDuck provides all of it with simply pointing your local database to connect via ATTACH 'md:' compared to directly reading from a local DuckDB database (duckdb path/to/file/db.duckdb) or parquet files (FROM nyc.parquet or FROM read_parquet('test.not-ending-with-parquet')) that only you have access to.

The simplest way is to initially upload data to MotherDuck once, and then have access to the data from anywhere (see example later in the article).

[!example] How to Upload data to MotherDuck from Local Storage

You can basically use different ways of synchronizing local DuckDB to MotherDuck via COPY FROM DATABASE, CREATE OR REPLACE DATABASE ... FROM '<path>', from Parquet files, to using Python files - you can find more details at sync-duckdb-to-motherduck.

Besides simply replacing local DuckDB with a data warehouse like MotherDuck, MotherDuck has implemented a specific architecture called dual execution. It’s built on top of their 1.5-Tier Architecture. It’s a novel 1.5-tier architecture powered by WebAssembly (Wasm). Unlike the more traditional 3-Tier architecture that operates between the client and server, the 1.5-tier directly returns the request in the client (browser), reducing latency for server requests and network round trips.

Traditional applications are built on a 3-Tier Architecture, which requires several intermediary operations to run between the end user interface, server, and underlying database. MotherDuck’s 1.5-tier architecture has the same DuckDB engine running inside the user’s web browser and in the cloud.

The developers can move data closer to the user to create analytics experiences that run instantly with the benefit of still scaling with MotherDuck as the backend. Check their CIDR paper on DuckDB in the cloud and in the client, on how this works in detail.

What Does the Dual Execution Do?

Since the initial paper, the dual execution has evolved and makes MotherDuck more than just “DuckDB in the cloud”. When you ATTACH 'md:' locally or in the web, you get a two-node distributed system that automatically routes query stages to wherever they run best.

An example with using dbt: In your sources.yml of dbt you can simply define dev with DuckDB and prod with MotherDuck like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

your-project:
	target: prod
	outputs:
		dev:
			type: duckdb
			schema: project_dev
			path: "path/locally.duckdb"
			thread: 1
		...
		prod:
			type: duckdb
			schema: project
			path: "md:prod_project"
			thread: 1
		...

Smaller data gets processed locally with millisecond response times and if needed, can extend to run in the cloud, using cross-environment joins that transfer only the necessary intermediate results. As a user, you won’t notice the difference in using simple DuckDB.

This uses your laptop’s power with DuckDB as a first-class compute node. As MotherDuck CEO Jordan Tigani put it: “Laptops these days are extremely powerful and you can get answers in a handful of milliseconds, whereas if you had to ask a cloud service, the initial request wouldn’t have even gotten there.”

In a way, MotherDuck is a lightweight alternative to Spark for single-node or moderately-sized analytical workloads (it does not support distributed, multi-node processing on massive datasets), but it’s far easier to set up, has no cluster management, and scales to terabytes. Without the setup cost or the operational burden for tasks that don’t need the massive scale Spark provides, you get an out-of-the-box data warehouse that handles scale very conveniently for us users.

[!example] One more advantage is multi-user collaboration

DuckDB is single-writer, and MotherDuck is what unlocks the “multiplayer” angle with its integrated notebooks (which are also available locally with duckdb --ui, but not shareable on the web).

Why Rill?

So what does Rill bring to the table, and why do they work so well with MotherDuck?

If MotherDuck gives you a data warehouse that feels like a local database, Rill gives you a BI tool that feels like a code editor getting you up and running with a single binary you can start with. The name Rill is from an old English word and meaning “stream”, and it has strong templates integrated that scaffold any BI requirements in seconds. Both MotherDuck and Rill are built on the conviction that focusing on developer experience will help data teams implement great data solutions that not only work, but are fun.

Rill’s core idea is simple: define your entire BI stack (data sources, SQL models, metrics (total_revenue: sum(amount)), dashboards) as YAML and SQL files in a Git repository. Start simply with one command (curl https://rill.sh | sh), run rill start, and you have a local development environment backed by an embedded DuckDB instance delivering sub-second queries. Push to Git, deploy to Rill Cloud, and your dashboards are live. The same declarative files, the same SQL, the same metrics, just a different runtime.

Moreover, this “BI-as-Code” approach turns out to be exactly what makes Rill a natural companion for agentic workflows, because all artifacts for data and BI are defined declaratively and locally, any agent can use them for context and build autonomously on top, while still letting the user verify that everything is correct and works by quickly running the Rill CLI locally or in a CI/CD pipeline.

Rill embeds DuckDB under the hood. Connecting it to MotherDuck requires nothing more than a YAML connector config pointing at md:my_database with a token property such as `token: “{{ .env.CONNECTOR_MOTHERDUCK_TOKEN }}”.

The SQL models are identical, meaning no syntax changes, no migration, no new query dialect to learn. The only thing that changes is where the data lives, but that has no effect on the user experience.

Rill was built around the idea of:

That instead of hiding business logic in a proprietary GUI that only humans can click through, you make it readable code that anyone, or anything, can openly read and reason about.

Rill with its YAML dashboards and a CLI-first workflow has positioned itself perfectly for working with AI agents. That wasn’t planned or foreseen, but it turns out that context is king and that tools designed for developer simplicity are exactly what agents need with readable definitions and fast deterministic engines.

The result is a stack where your metrics are a source of truth you can version, audit, and feed directly into an agent’s context window, and where switching from a local DuckDB file to a serverless cloud warehouse is a one-line change.

Metrics and SQL

We all probably agree that metrics are the key to codify business knowledge, giving us all the benefits of a software design approach (versioned, automatable, testable, etc.) and automating metrics with an agent, while still being able to define complex metrics ourselves if needed. And an agentic-friendly environment where agents get their concise context and collaborate with a domain expert and human in the loop.

It’s good to know that Rill provides an integrated Metrics Layer. It’s excellent as it’s just a YAML file too. Meaning you can integrate it into other notebooks, or data apps easily, but also just with Rill for building multiple dashboards, conversational analytics, and canvas on top of a unified metrics layer.

Instead of integrating complex metrics multiple times in different dashboards, we can just reference the metrics layer. Besides the metrics, we need fast, instant responses, even more so when we let agents work autonomously. We can’t wait 5-10 minutes for a simple question until all research through the agents is done.

[!info] Metrics SQL: A language built on top of SQL

Additionally, Rill offers a dedicated query language that extends on the strength of SQL, called Metrics SQL. It’s a dedicated SQL dialect designed for querying data from Metrics Views.

I wrote more at a SQL-Based Semantic Layer, but it helps to simplify your SQL as it empowers the SQL language, or learn more about the philosophy from Mike, the CEO of Rill, in an insightful podcast with Joe Reis where they talk about the future of dashboards and how agents and navigating the new era of BI and analytics works.

Conversational BI: Rill Turns Dashboards into Code (and Code into Agent-readable context)

When looking at Conversational BI and its benefits, we can say that “Conversations can generate code, and code generates insights”. Code is the best abstraction, and with agents, we can easily make it available to non-programmers.

Code as the abstraction layer in most cases. But why? Because if you create a hard-coded interface language, or an API, you can only do what you need. With code (usually Python, or SQL in this case too), we can do much more. We can use all the functions of the language versus the implemented API. It’s easier to maintain, and also automate.

RudderStack reinforced this narrative from the infrastructure side:

Most of today’s commercial data tools are designed for humans, not for automation. Their primary interfaces are web dashboards, which are convenient for analysts, but opaque to code.

This means if we want agent tools to analyze our code base, we need to let them access our code, or in the case of Rill, declaratively defined dashboards, metrics in the metrics layer, and data sources.

We can do even more when we use the chat interface to interact, making it usable for humans again, making it usable for humans again by using natural language as the primary interface. That’s where Rill offers an extensive integration with agent workflows through generating a dashboard based on existing sources, models, and defined metrics in the metrics layer:

These features are also integrated as Conversational BI, letting you explore your business numbers with the interface of a natural language and chat. With Cursor and agentic code-like suggestions, but referring to pre-defined metrics for asking specific questions:

Here’s a full video of what you can do with it.

The chat interface provides charts that can be further explored or integrated into your dashboards. What I like most is that in the responses, you can just click on them (e.g. 1.) and it will appear as a pivot table in (2.) where you can dig into more details by adding more dimensions and metrics:

Showcase of how interactive dashboards are created on the fly, to explore and open in a pivot table directly inside Rill. The links are clickable, see a short video of how that looks in action:

[!note] You can also use Claude integration with MCP

Rill’s MCP can also be used, see Showcase of Rill MCP with Claude Desktop - YouTube. Or Cursor if that is your preferred AI-based IDE at BI-as-Code and the New Era of GenBI Demo - YouTube.

Let’s now look at different analytics use cases and implementations with these handy features combined with MotherDuck as the backend.

MotherDuck + Rill in Action: Three Examples

Let’s now look at actual implementations with these features combined. We’ll start with how to connect Rill to MotherDuck, then walk through two open-source examples you can try yourself, and finish with a real-world customer showcase.

Connecting Rill to MotherDuck

Since Rill already embeds DuckDB, connecting to MotherDuck requires only a four-line YAML connector config with a token and a md: path. Add a motherduck.yaml to your connectors/ folder:

1
2
3
4
5

type: connector
driver: duckdb
mode: readwrite
token: "{{ .env.CONNECTOR_MOTHERDUCK_TOKEN }}"
path: "md:my_database"

Compare that to a local DuckDB connector:

1
2
3

type: connector
driver: duckdb
dsn: "my_database.duckdb"

The only difference is token + path: "md:..." instead of dsn. Set the token as an environment variable (see Rill docs for details), and your SQL models, metrics, and dashboards work identically — whether the data lives on your laptop or in MotherDuck’s cloud.

In 2025, Rill significantly strengthened its native connectivity to enable zero-copy, blazingly fast analytics without moving data, making this connection even more seamless.

Stack Overflow Developer Survey: Zero-Pipeline Analytics

The simplest way to experience MotherDuck + Rill is with data you already have. Every free MotherDuck account ships with sample_data.stackoverflow_survey.survey_results² — 600k+ professional developer responses from 2019-2024. No ETL needed.

The motherduck-rill project builds a complete analytics stack on this data: 4 SQL models (staging, technology usage, developer profiles, database analysis), 3 metrics views with 17+ measures, and 3 canvas dashboards — all as pure SQL + YAML in a Git repo. No Python, no orchestrator, no data pipeline.

1
2
3
4
5

git clone https://github.com/sspaeti/motherduck-rill.git && cd motherduck-rill
cp .env.example .env
# Add your MotherDuck token to .env
rill start
# Open http://localhost:9009

That’s it. One command and you’re exploring which databases are most desired in the US, which languages pay the highest salaries, or how AI adoption shifted across years — all backed by MotherDuck’s serverless cloud.

Answering the question of “Which databases are most desired in the US according to Stack Overflow”:

The SQL models use standard DuckDB syntax throughout. For example, the database_analysis model unnests semicolon-separated survey responses into one row per database per relationship type (used, admired, desired), then the metrics view aggregates them with COUNT(DISTINCT ResponseId). The same SQL runs locally via embedded DuckDB or via MotherDuck — the connector config is the only difference. This is what BI-as-Code looks like in practice.

The above example could be created based on a couple of simple prompts, as Rill’s definitions are all local and the data accessible through MotherDuck through DuckDB CLI.

Multi-Cloud Cost Analyzer: Production-Grade Connector Switching

For a more production-like setup, I use the cloud-cost-analyzer I’ve built in a previous edition to visualize your costs from different hyperscalers with ClickHouse, and now added MotherDuck.

This project shows how MotherDuck fits into a real data pipeline alongside local DuckDB and ClickHouse. Same dlt pipelines, same Rill dashboards where I just added a different destination to dlt. Zero pipeline code changes were needed.

The key insight: MotherDuck uses DuckDB SQL syntax, so the SQL models share all functions with local DuckDB. Only the FROM clause differs — read_parquet('...') locally vs schema.table on MotherDuck. The Rill SQL models use a 3-way conditional to switch to work around these small differences between connectors:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

{{ if eq .env.RILL_CONNECTOR "motherduck" }}
SELECT CAST(SPLIT_PART(identity_time_interval, 'T', 1) AS DATE) AS date, *
FROM aws_costs.cur_export_test_00001

{{ else if eq .env.RILL_CONNECTOR "clickhouse" }}
SELECT toDate(splitByChar('T', identity_time_interval)[1]) AS date, *
FROM aws_costs___cur_export_test_00001

{{ else }}
SELECT CAST(SPLIT_PART(identity_time_interval, 'T', 1) AS DATE) AS date, *
FROM read_parquet('data/aws_costs/cur_export_test_00001/*.parquet')

{{ end }}

Notice how MotherDuck and local DuckDB share the exact same SQL functions (CAST, SPLIT_PART) — only the FROM source changes. ClickHouse needs its own dialect (toDate, splitByChar). This is the practical advantage of MotherDuck being DuckDB in the cloud: your SQL models stay the same.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16

git clone git@github.com:ssp-data/cloud-cost-analyzer.git
cd cloud-cost-analyzer
uv sync

# 1. Add your MotherDuck token to .dlt/secrets.toml
#    [destination.motherduck.credentials]
#    password = "eyJ..."

# 2. Add token to viz_rill/.env
#    CONNECTOR_MOTHERDUCK_TOKEN="eyJ..."

# 3. Load data into MotherDuck (same pipelines, different destination)
make run-etl-motherduck

# 4. Start Rill dashboards backed by MotherDuck
make serve-motherduck

Under the hood, DLT_DESTINATION=motherduck tells dlt to write to MotherDuck instead of local parquet files. The same data is then also visible in the MotherDuck web UI for ad-hoc querying alongside the Rill dashboards.

This pattern — make serve-motherduck vs make serve vs make serve-clickhouse — shows what switching from local to cloud looks like in a CLI-first stack.

Driotech: Agentic Analytics in Production

Beyond my own demos, Salomon from Driotech showcased how they use Rill for agentic analytics with client data using MotherDuck in combination with Airbyte, dlt, BigQuery and dbt.

In his webinar on empowering businesses with agentic analytics, he walked through a B2B sales use case that highlights exactly the principles we discussed above.

His key takeaways align perfectly with the MotherDuck + Rill thesis:

1. Metrics as the foundation: Before any AI agent can work reliably, you need clearly defined KPIs and a single source of truth. “If you feed any AI agent with a mess, you’re going to end up with an even bigger mess.” → This is exactly what Rill’s metrics layer provides: versioned, agreed-upon definitions in YAML.
2. Governance as guardrails: The agent doesn’t hallucinate on business concepts because the semantic model constrains what it can query. When asking “what are our shipping costs?” → The agent looks up the exact metric definition, avoiding guessing from raw data.
3. From reactive to proactive: Beyond just answering questions, the demo showed creating alerts (“if customer orders drop to zero in 14 days, email the account manager”) and scheduled reports → Pushing insights to stakeholders automatically.
4. Code-defined dashboards: Salomon explicitly called out that Rill’s code-based approach means “AI agents are able to build the dashboards for us” → Because the language of AI is code, and Rill’s dashboards are just that.

The above webinar reinforces a central point we’ve discussed above that conversational BI can be used today in Rill. When you have a solid semantic foundation (metrics in Rill), a scalable backend (MotherDuck), and a code-first workflow, the agentic layer becomes practical, trustworthy, and something you can deploy to real users today.

What’s Next, Dashboards no More?

Seeing these examples, and where we are heading with agentic BI, one might ask now, do we even still need dashboards? We can just ask the chatbot?

I agree with Mike’s argument that: “No, well-crafted, and especially operational dashboards, they will never go away”. Because a visualization can provide so much more condensed information in a couple of seconds that a chat never will, or would require many back-and-forth chats.

There’s also a difference between “a” dashboard and the dashboards. E.g. more than half of dashboards are just exploratory in nature to quickly explore the business or data. Additionally, the sales dashboards that someone spent weeks or months, sometimes years for large companies, to perfect and ensure the correct numbers, deriving key decisions for the business from it or defining a sales rep’s salary as how much they sold usually is tracked in a dashboard as well.

One more thing that chatbots can’t replace is so-called “drilling down”, and using data as a REPL with pivot tables - as they never die. Quickly drilling down to the lowest details and back up to the aggregated data within seconds, simply dragging and dropping dimensions and measures around.

[!tip] People don’t know what they are looking for
Benn Stancil argued, “the challenge with data exploration is not that people don’t have the ability to manipulate data; it’s that they don’t know what they’re looking for.”

Natural Language Interface: Convenient, but Inaccurate?

With agents, we can use natural language as an interface to input domain knowledge, and agents will do the technical translation and implement it in such a code and declarative-first approach, where the context is clearly and distinctly defined - much more than natural language, which contains lots of nuances and ambiguity.

So with that approach, the agent will put it into deterministic YAML, that can then be reviewed, tested and automated against. So we move from Human to agents to context to iteration and finally, visualization:

Similar to what we discussed in GenBI, iterating much faster than with the traditional, non-generative way:

Self-Serve with BI-as-Code providing the Context

So what’s next? Are we arriving at self-service BI finally? (the never-ending promise🙂).

Agents with natural language solve a big problem that self-serve always strived for: Giving each domain user who is less technical an edge to do self-serve themselves, and potentially even go further and fix the data by prompting the data pipeline to fix the correct timestamp, or update a dbt model with the right table source.

With domain experts doing more developer work with agents, we can combine domain knowledge with coding abilities of agents, bridged by natural language with BI-as-Code providing the semantic context that includes models, metrics and even dashboards in plain YAML.

Context is king for the near future. Everything that can be locally defined, such as Rill’s metrics layer and dashboards, will be so much faster and better built with agents.

But BI is not the only domain noticing this power, Kurt Buhler of Tabular Editor wrote: “CLI tools provide an alternative way to interact with software in a terminal by writing and executing commands. This command-line interface is very suitable for agents.”

SQL, YAML: And why Language Choices Matter

Also, with Rill and MotherDuck we choose SQL as our primary language and interface and YAML as structured format to store. And the language choice matters because the training data for widely adopted languages like SQL is larger, thus LLMs are better at generating BI-as-code in SQL than DAX, LookML, or some obscure language.

Wes McKinney even argues that AI agents are enabling him to build software in languages like Go and Swift, despite his lack of prior experience. He says that “human ergonomics in programming languages matters much less now,” as agents prioritize fast compile-test cycles and frictionless distribution, favoring languages like Go and Rust over Python for new systems. Interested in more, check the conversation with Wes and Mike.

Limitations

The one limitation for the future of data might be the imprecise way of natural language and how we communicate. For example: “give me the analytics for this week?” Did you mean “from today until last week”? Or “full weeks Monday to Sunday”? Or “starting from midnight, or during the day”? So many unknowns and misinterpretations possible.

The other is that the future of data needs to be deterministic and reproducible, to backfill faulty data, but AI agents are the opposite. And that can be challenging.

Text-Based, Local-First: The Architecture Agents Need

The connectors between source and destination are getting more flexible, more fluid, self-healing. Knowledge workers might soon be able to not only figure out the problem, but also act on it directly: “send an email to…” to fix the problem.

As the examples in this article show, we’ve come a long way with Self-Serve BI, and we might already be there. Keep in mind the three pillars: semantics, stewardship, speed as you work in the Agentic Era.

The MotherDuck + Rill story is ultimately about the data industry discovering that the tools best suited for AI agents are the same tools that respect simplicity, transparency, and developer ergonomics.

The “small data” thesis didn’t anticipate the AI agent revolution, but it created the conditions for it: when your data fits on a laptop and your dashboards are YAML files, an AI agent can read, reason about, and act on your entire analytics stack.

The irony is that going back to local-first, text-based, SQL-defined analytics turns out to be the most forward-looking architecture. And dashboards become agents when they’re written as code.

–

If any of these interest you more, also check out related articles around conversational BI, BI as code, and how AI can self-serve us in the world of BI:

• BI-as-Code and the New Era of GenBI
• Data Modeling for the Agentic Era: Semantics, Speed, and Stewardship
• Has Self-Serve BI Finally Arrived Thanks to AI?

Full article published at Rilldata.com - written as part of my services

My prediction, and hope, is that the [[Future of Blogging]] is more connected. Not only one dimensional, like a single sheet of paper, but think of a maze, where you can go in, explore new things to learn.

This is how I built up my Second Brain, and you can see the interactive graph at the end of this blog, connecting all notes and blogs that are related.

This article is based on a recent interview about “Write that Blog”. This triggered me to finally write this piece after collecting 100s of notes related to writing online and blogging in my second brain.

Why I Started Blogging: Learning OSS Tools

A quick note on how I got started. Mainly it was out of curiosity. As a business intelligence specialist with a Microsoft licence, I was more curious about open-source tools that had similar abilities as [[SSAS]], [[SSRS]] that were used at work, even more so, the programmatic first approach to automate things, instead of clicking myself through the UI of older GUI first approaches.

I had some when I lived in Copenhagen, Denmark, which I used to explore and document what I learned. As I already had a domain (sspaeti.com) and experience in web development with weekly party pictures that I ran for many years, but wasn’t active anymore as Facebook and other portals got created, I decided to pivot it to a personal blog - which was very popular back then.

So I started with my WordPress blog and uploaded some scripts and learnings around Microsoft and related automation I learned, and was re-using often. Then I did a deep dive and a series on data warehouse automation tools, which got very good feedback after the initial blogs didn’t go anywhere.

I found myself enjoying the process of distilling knowledge in a compact format, so others, and mainly myself, could learn new topics. The [[Feedback Loop]] was another amazing feeling that I didn’t know beforehand, along the principle [[The more you share the more you get]] - as people were giving me suggestions, new ideas, sometimes criticism. But all to find even more open source tools and interesting approaches.

What Started as a Hobby Turned into a Full-time Job and Business

Writing became one of my favorite hobbies, and I got lots of fulfillment, not the short term dopamine hit, but the long-term [[Deep Happiness]] of learning, getting appreciated by readers, and the process of turning my long taken notes into something more usable for people to share. [[Learn in Public|Learning in Public]] as some called it later on.

I reserved Friday nights in my favorite library in Copenhagen, bought my favorite coffee at Espresso House, mostly a nice cookie or something sweet, and then off for 2-4 hours. Sometimes nothing really good came out, it was hard. Other times I was just trying new tools like [[Dagster]], [[Delta Lake]] etc., and others I was in [[Deep Work|deep flow]] of writing, almost like trance.

The breakthrough came much later, three years in, when I wrote about a new upcoming topic, and how the transition from data warehouse I see, called Data Engineering, the future of Data Warehousing?. This was the first viral post, and popular figures like Dan Linstedt commented on it. It was surreal back then, why would these people read my article?

But it gave me the motivation to continue. Sure I love writing and sharing in public, but not sure if I wouldn’t have people reading it, if I would have continued until today.

[!note] A short journey of how my domain and website evolved

I started my first blog in 2015 - but I was online and registered a domain in 2004. I bought the domain sspaeti.com where my first endeavor was web development with HTML, CSS and PHP. The classic Apache years (fun fact, I still deploy to apache server to this day, but it’s only static HTMLs today :)

From 2005-2014 I ran a local forum and party guide (this was before FB :) and then in 2015 my first data-related post. 2016-2018: Regular blogging on Business Intelligence and data topics. 2019 I started to focus more on open-source data engineering.
2021 I moved from WordPress to GoHugo and 2022 I added the second brain to my website which meant all my notes and blogs were powered by Markdown which led me to share much more as it took me no conversion or work to publish anymore. What I wrote, I could just publish as is on my second brain. To this day, I have ~9000 private notes and ~1000 public notes. And 81 blog posts and some chapters of an early book I’m writing in Markdown too :).

2023 I changed the domain to ssp.sh, as it’s shorter :)

How Did I Manage to Continue to This Day?

[[Writing is hard]] as anyone will tell who does it. So why do I torture myself to do it to this day? Even made it my full time work, as I’m currently self-employed and work as a full time author.

The answer is not straightforward, but to say the truth, I still love it to this day. Writing words is my canvas as an artist, where I can let out my thoughts, be creative, bring something complex into simple terms. Into something that anyone might want to read.

How Has Blogging Changed over the Last 10 Years?

During my start, where I created a WordPress personal blog website, to today, there have been different evolutions, but overall, not much has changed in terms of personal blogs.

These are still the same, except that we changed the technology a couple of times, from using Flash websites to hand writing PHP/HTML/MySQL to using WordPress to Medium and [[Static Site Generators (SSG)]] to Substack today, the main change is social media. Before, personal blogs had more authority. Everyone was linking to other blogs, currently a couple of social media tech giants have the monopoly and you almost need to share there to be discovered.

When I started, I used Twitter and LinkedIn already too, but “the game” of distribution has changed. But again, the sole purpose of personal blogs is the same.

Today with AI we are even in a new era, with all the “AI Slop” generated and shared all over the place. I believe, and see it as my work as a professional writer, that the craft of writing, and [[Writing Manually]], gets even more important.

Writing is communication, and we can’t communicate through a filter, which at the moment many are doing with converting bullets into prose and the reader summarizes from prose to bullet points - watering down the actual points and wording the original author has made. To the point where most people, me included, [[I’d rather Read the Prompt|would rather read the prompt]].

Blogs vs. Second Brain Notes

One approach that I like to push, and many are doing locally with [[Obsidian]], is connected note taking. I shared my Obsidian notes that are worth sharing on my public second brain (find my process at [[Public Second Brain with Quartz]] of adding #publish and it will be on my site, no conversion needed, the code and utilities are shared on GitHub, too)

Bringing back connected personal notes, but also internally on your website - using synergies between your blog and second brain. The way I think as of now about [[Sharing as Second Brain Note vs a Blog Post]]:

The second brain helps me to share whatever is in my mind, and the blog helps me to refine. Notes compound and always evolving. Blog posts capture a moment in time.

There’s also the difference between long-term, always updated, and [[compounding]] notes vs. the one time distilled blog article. They work so well together. As you might notice, most of my links in this article, with much more information, are long-term notes that I’m collecting and refining over the years, linked to my second brain.

This way, I can bring all notes into one storyline, the way I’m currently thinking, sharing it in the form of a blog, as this one, while continually updating the related notes all linked here on long-term strategy for blogging, and with its different [[Type of Notes]].

Connects Knowledge: Helping Learning the Same way as Our Brain Does

I’m thinking of Designing Data-Intensive Applications by Martin Kleppmann), where he added maps to his book, that correlated similar terms:

I see the connected, interactive graph on my second brain, and on my book (just recently added) the same way. It helps learning.

It’s proven that we learn much better if we can associate to an existing term or something we know, versus something new that is orphaned in our brain, without a connection and synapse to another thought (or note in our case). It’s hard to remember and learn from it.

In a [[Second Brain]] and [[Digital Garden]] approach, you connect every note at least to one existing term. I also like to add its origin so I always know where it came from. More on My Obsidian Note-Taking Workflow if that interests you more.

For example the below note about [[Functional Data Engineering]] (← click here to see the graph and backlinks in action) shows how besides the written text, you can glance connected notes through the interactive graph or through backlinks.
![[img_index.en_1772811747298.webp]]

We can visually see things that are otherwise almost impossible to grasp or see. Like a map of a city can convey information density that no chat or explanation can do by explaining to someone on the phone or in written text. It’s the same with the graph.

And the best part, it’s additional, so you don’t need to look at it at all. But most helpful when you want to learn or might not know the space that well yet, you can see a term or connection you know, and immediately connect your brain, that these belong together, probably remember forever, or much longer.

E.g. in the above example, we might see that functional data engineering is linked to clarity, and to [[No Less Code vs Code|Code Is Still the Best Abstraction]], which might be non-obvious, but really helpful to know.

Again, linked notes are the best way to organize knowledge, especially optimized for learning. Knowledge doesn’t grow linearly. It expands as a network over different seasons. I write more about that phenomenon and continue to update at [[Future of Blogging]].

The Process - And the Difference between Long-Term and Short-Term

My process is essentially:

1. idea occurred by reading a book, listening to a podcast, talking to someone or else
2. writing it down in my private second brain
3. connect to existing notes, refine the note, add new thoughts and notes
4. eventually add #publish and publish online on my public second brain
5. eventually distill that note with many others and write a blog post about a specific and related topic of that note
6. continue updating and refining the note
7. eventually writing another blog that relates to it, using that improved note
8. using the note for my book topics
9. refining note

I think you get the gist. The initial one liner, the note that just existed based on a real insight, is the actual most important information of the whole process, in my opinion.

Not to say that the blog articles are not, but they both need each other. With writing the blog, I distill and connect multiple notes at a current timestamp into a frozen article. During that process, notes also get updated. And while sharing the blog post, I get a lot of feedback, [[Feedback Loop]], which I then instead of adding to the blog, which I can’t as it’s a snapshot in time, add to the existing note.

So feedback is actively processed, and massaged into my private or public second brain, improving my overall approach. I think this process of connected notes from private, to public note to blog and continued note, is even more powerful than Niklas Luhmann’s [[Zettelkasten]], that revolutionized the [[Smart Note Taking]] approach based on zettels, small unique ideas that he connected to others.

[!info] No Research needed with this process

Related is also that through this process, [[Why I don’t Research|I don’t need to research]] in a classical sense for topics. As my life and insights come steadily in, and get massaged and integrated like a slow flowing river, all organically.

The Form of Writing: long Form or short

My writing usually tends to be very long-form. Because I take lots of notes that I try to connect and write in an interesting way, I tend to get very long. Same as this writing already is, but I still have 3000+ notes collected to go.

Also, long-form writing evokes a deeper relationship and trust that is hard to captivate with a couple of words. That’s also why a [[Reading Books for a Happy Life|book]] can connect you to the author like no other medium can.

It’s also a question of long term game and writing for it to stay relevant for many years to come, or just capturing a quick trend and harshly (fast?) putting out a blog. These are totally different forms of content, and strategies. The latter usually also used on social media, to create big attention to go viral for a short term.

But what I always ask myself, what’s the gain from it? Ultimately, likes and followers are just a [[Vanity Metric]], and to me at least, don’t count as much as a real human reading these words. Not leaving a like or comment, but just having made a connection or an impact on someone in another part of the world I don’t know (yet? I’m always happy to get introduction emails from my readers! :)). Or just inspiring or making you think about something related, or just learning something new.

That’s at least my main goal. There’s no hidden goal or message behind my writing. Obviously if I write for my clients, it’s a bit different, as I want them to succeed, whereas I write for myself, I just want to let out my thoughts. But what I learned over the years is that [[Writing from The heart]], being genuine, is also helping for work related topics, as at the end of the day, it’s still a human being reading it - and therefore the same principle applies as if it were just a random blog post.

My Writing Process

I’d like now to switch gears a bit as we went through the differentiation of compounding, refined notes and in-time blog posts, and talk about my writing process that I have mastered or improved over the years.

This is my unique writing approach and even more so, note-taking approach. This is not how you have to do it, and probably won’t work for you. As I learned on the How I Write (Podcast) by David Perell, each author has a totally different approach. And this here is mine.

But here I want to share a little bit more about my strategies, my approach to writing, and my tips and tricks I have learned and noted down over the years.

I Spend Many Hours, Weeks and Months on Single Blog Posts

But even before that, a quick prefix: I spend many hours, weeks, sometimes months on a single blog post. Why Are We Here on Earth? for example, I wrote over the course of two years - but if you include my note taking, it sometimes is obviously even longer, because some notes of mine are ten years and older.

There’s also the longer I work on it, the more learning, and sometimes struggles I can put into a piece, which helps the piece to not get outdated the next months too. Something that I’m grappling with over months and years most probably won’t be gone tomorrow.

This is one reason why I don’t like to write too much about AI at the current pace, everything I write, and would spend lots of hours might be outdated the moment I publish.

That’s why my approach is just collecting and refining my thoughts on a second brain note, for example in this case on [[Will AI replace Humans]] and related notes - which made it already on the frontpage of Hackernews - but I’m sure at some point I will take that note and all its relevant related notes, and will distill into a single blog post. But the time hasn’t come yet, as so much is changing.

But it’s not that I don’t do it at all, sometimes I will write something quick, something maybe less long-term, but usually, it’s just less fun for me to write, and maybe less challenging? Although, some topics and articles that I have slept over too long just poured out of me in one go, no sophisticated linking in my second brain or other approach, just a blank page and writing it down. But usually these are topics that I have read extensively about, I’m discussing with people or are just dear to my heart, that my subconscious is just working on it until it’s telling me it’s ready, and then I must not miss the opportunity and just write it down.

A little similar to this piece and topic. It’s so dear to my heart, and something I wanted to write for so long, that I have never done it, and now with the “write that blog” interview, it triggered so many questions that I just went on and wrote until now in one flow. No breaks, just free flow and combining different notes I have in my Obsidian vault.

This is how my vault and process looks right now, with:

1. The current note
2. Is the long long outline (you can’t even see half of it),
3. Is related notes through smart connections
4. Is the initial write that blog interview I answered
5. Is another connected note I have written just above about
6. These are more related notes
7. And you see word counts on the lower right and Vim Motions I write the article in

[!example] Don’t focus too much on the numbers, but on writing
With social media, you could focus too much on [[Vanity Metric]], and how many likes you get on social. But I try not to give too much about it, though it’s still needed. I wrote more about Well Being in Times of Algorithms, my personal essay towards a better World Wide Web, and how well-being is connected to social media.

Ultimate Goal: Good Storytelling

If I had to summarize my writing process, or the goal of it, then it crystallized to me lately that the goal is to have the ultimate storytelling. I want to write about a topic that has an intro that catches the attention, then has a great body, and finishes with a hook and ties everything together.

Storytelling like in the movies, it’s true for writing too, where they have the main act, second act, the villain etc. But with the difference, you can’t use fancy show effects, you are left with simple words.

[!abstract] What is Good Storytelling?

This obviously is very personal, and differs from person to person. To me, most of it boils down to the art of leaving things out, which I am getting much better at over time. And I think that is really what storytelling is all about.

That’s why it’s very important to use what you have as a writer. In writing a blog like this, one of the most important and one I like to use most, it’s the length of a paragraph, making them look good, break at the right timing when re-reading. End on a high, start the next that connects but with a new insight. The paragraphs should be of different lengths, they should be interesting, and change over time.

Mix it up with images, with some quotes or what I like a lot, [[Admonition (Call-outs)|callouts]]. The reason why I like callouts is I can add an additional story, a side note in a way to not distract from the main storyline, but I can serve some readers who like some behind the scenes or more information. Plus they look beautiful in my eyes, adding different colors to the blog post as each of my different types of callout has a different color. It makes it more interesting aesthetically, and that also helps to want to read something more in my opinion, the aesthetic can help big time.

Leave with a Spark✨. Making it Interesting

Besides the ultimate goal of having a good storyline, having a common thread, a nice reading flow and outline is key to keep you, the reader, engaged. I like to jump a bit around.

Not only cutting some topics short, moving on to something else, maybe coming back, maybe not, leaving the reader in the blank, making it more interesting. The key of good writing is leaving out what needs to be left out. It’s an artform, because I could ramble forever on this topic, as I’m super passionate about it - as you might have noticed - but I need to always keep in mind to not bore you. To give you new insights.

That’s why I’m switching now to making it interesting. Besides switching from topic to topic, I also like to go very deep in a topic, and then zooming out very high-level, only to go very deep again in the next sentence.

Zooming in and out helps the reader to not lose the overview, but also learning something new. I usually don’t spend too much time in the middle “zooming level”, this section is boring to me as it’s too vague (not detailed and concrete, and not guiding with not enough overview).

Switching all the time might feel a little like a rollercoaster, but rollercoasters are fun, so do I envision my articles. Writing and its process boils down to me to:

[[Writing from The heart]] is the best. True, honest, and genuine human-to-human communication.

Writing from Curiosity

My best writing comes from my own curiosity. I want to answer a question for myself, even better if I don’t know the answer beforehand. Not knowing where I’m heading to.

I usually set a title, and then go with the flow, see where it leads me—these are the best writings of mine. If I have to write about a certain topic, or outline, it couldn’t be more boring, and that’s usually reflected in my writing too.

The exception is if I know the space very well, I can write a leadership thought piece, bringing together 20 years into one blog. The challenge is again in nailing the storytelling part to make twenty years coherent from start to end.

My Writing Style

Finding your writing style and writing voice is something that was very hard for me. But I think is key to become a writer, especially a professional writer.

It takes time. What helped me for sure, was to read many books, finding my favorite authors and identify their writing style. This helped me to find that I liked Derek Sivers books. Initially, I didn’t know why, until I found more about his writing style, his personality in interviews, read more from him, and analyzed his work.

I found that his minimalistic style, to scrap each unneeded word, straight to the point and providing value while inspiring with different takes that you haven’t heard already a hundred times. He writes genuinely, he also [[Journaling|journals]] 3-4 hours almost daily, thinking and brainstorming a lot in his head and second brain.

That’s also what led me to journal and write in my second brain, like a physician, experiment with different formulas and ideas, to see what comes out. That’s my second brain idea creation. I have the two phases, the idea creation and finishing part. Again, the second brain is where I start with a one-liner, a note from a friend, listing interesting things, linking to existing notes.

Later when distilling into a blog post, or sharing a public second brain note, I will tackle the deeper meaning, the connection with other ideas and areas of my life or things I’m currently learning or have learned a long time ago.

I try not to force it. How many times have I tried to force it, only to go to bed early, and the next day wake up and just have it flow out of my fingers.

Writing is Rewriting

Most of it is also just rewriting. When you write something 3-4 times, when you sleep over it, your subconscious has worked on it while you walk, it always gets better.

It’s a way of personal editing process. It’s also a way of writing style. Jason Fried and Haruki Murakami from Novelist as a Vocation (an amazing book for writers), are constantly re-writing. Sometimes based on feedback of readers, sometimes based on a [[Gut Feeling]].

[!example] A short story from the book Novelist as a Vocation
Haruki Murakami writes in his book that once he lost a manuscript of a chapter. He was devastated, but had no other choice but to rewrite it.

Years later he found the manuscript again and was afraid it would be better than what he had handed in for his book that was already published. But the fear was all wrong, it was so much worse, he writes.

Finding Your Voice, but How?

So, how do you find your voice?

After all the different ways I wrote, I read from other authors, there is no one way, and I can’t tell you how yours will be, other than you start writing, and trusting in the process.

For me, my writing voice I defined as written in personal first person voice. Something I have experienced I can easily explain or write about. But making up a fake story, something they let you do in school, is something I was never good at.

I try to be authentic, friendly and succinct, with the goal of adding value to the readers.
I’m trying to give clues and tools concrete and extremely specific but also leave things out. Because I’m not an academic and I can only make suggestions about what I learned but cannot solve all the problems I write about. I also try not to overly copy others’ ideas, but make them my own through connecting and my own unique life experience. Including hardship and daily struggle and just life.

Key is also that you share in public. Keep writing until you find your voice. From there on, it will be much easier.

How I Found My Writing Voice: Through the English Language

I did live abroad for almost three years in a foreign country, learning Danish, but even more so English. And what happened there is what I would never have predicted.

The more I learned the language of English (I wasn’t fluent before), I started to read more books. I noticed that there are so many more books that weren’t available before, when I only read in German.

Also, I found out that I really liked the English language, the simpler grammar compared to High German as we like to call it in Switzerland. I found that I can express myself much better, more precisely as English has so many words for almost the same meaning, so you can choose and pick one that exactly describes what you want to say. Whereas in German I felt I always need to write a full novel to explain a simple thing very specifically. This might be good for fiction, but not for my technical writing, or also what I write here.

All of a sudden I was reading books all my free time, listening to Tim Ferriss and all his guests on his podcasts, and learning every day. Also, that was the time when I went to the library in Copenhagen and started writing, in English, a secondary language I was just about to learn properly (I had English in school before) and could converse and have small talk. I wrote more about my journey and about Finding My Pathless Path if you are curious to know more about that.

Simple English

As you know now, and might get from my grammatical errors here and there, my English isn’t my mother tongue.

For a long time I saw that as a disadvantage, but lately I figured that it might even be a strength of my writing. With my somewhat limited English vocabulary and language skills, leading to my articles and writing being much simpler English.

And one thing I learned over the years, the easier you can explain complex topics, and make it approachable, the easier for my reader to follow along. Also it makes the writer more approachable, less “snobbish” maybe?

And this is an advantage. My writing is much more approachable this way. It adds a natural constraint to my writing and makes my writing process potentially easier, that I am not even aware of during the writing, but helps me in a certain way I do write.

The Writer’s Toolkit: And the Tools I Use

Let’s come to the last bigger part of this already long article, the tools and methods I use.

My main tool is writing in an open format, that is just [[Markdown]] and then using [[Obsidian]] as the editor to connect these simple notes together in a meaningful way.

The Different Modes of My Writings with ‘Vim Motions’

Apart from that, the way I write is with something called [[Vim Language (and Motions)|Vim Motions]]. I have written extensively about it, and you might think these matter not so much.

I hope, if you write online, or program for a living, that you learned touch typing at some point in your life. If you have, you’d agree that it tremendously helped you with everything working on the computer, right? Not needing to see each key before you press.

Vim motions go a step further, essentially making each key on your keyboard a tool. In the default mode, when opening vim, each key press is doing a function. E.g. g is for jumping around (gg is jumping to the top of the document, ctrl + o jumps back where you left before. G jumps to the end of the document. $ jumps to the end of a line. And so on, I could go on forever, but if you want to actually write something, you’d need to switch to “insert-mode” with i (there’s also a for append or o), but you have different modes. I wrote in [[Four Modes of Writing]] I have four modes with vim motions at all times:

1. NORMAL mode: jumping around, reading, learning
2. INSERT mode: writing, thinking, making connections
3. VISUAL mode: copying, highlighting, format, designing
4. COMMAND mode: automate, fix, macros.

These vim motions, which are different from [[vim]] or [[Neovim]], the editor, are also available in Obsidian and almost any editor you know. Even Gmail has shortcuts like j to go down or k to go up, two common ways of navigation in vim motions. Even more, vim has a language, the vim language. This is super helpful as you don’t need to memorize 1000s of commands by heart, but can combine them. Almost like Streetfighter where you can do a combo.

Besides vim motions, which I write much more on Why Vim Is More than Just an Editor, you can edit at the precision of a surgeon. If interested, also check out my video on [[Vim Motions for Writers]], where I made a timelapse of how that looks:

[!tip] A Trick you can use: Writing within the App vs. a Note app

Sometimes it’s hard to write in your notes app or offline, but if you send an email to a friend, or write in a LinkedIn post or in your blog editor, the pressure is on. You know it’s going to go live, or it’s for a certain friend. This can help you unblock [[Writing is Hard|writer’s block]] or produce better quality.

The Medium

Different mediums to publish, to write on, to take notes are out there, when to use which?

The Medium to Publish

Which medium, which platform do you use? Nowadays you have many options, Substack, Medium, Ghost or other [[Open Subscription Platforms]].

I would always recommend having your own domain. If you like web design and tinkering a bit, even more now with [[Vibe Code Agents|AI Agents Tools]], you should start with a [[Static Site Generators (SSG)]].

These allow you to use Markdown as the format, and owning your content, not losing backlinks when switching platform, and building the domain ranking over time, leading to more authority when searching for a job, or your own business or side projects. Or also just a hobby where you share learnings and topics of interest to you.

[!example] Medium in Taking Notes
I wrote much more on [[Digital vs Paper]], where I explain that I use both - and also share examples of how that process from physical paper notes to digital notes can look like.

Tools: Laptops, (distraction-free) Typewriter, Unitaskers!

An important part is also to make it fun! I do that with different devices. Recently I even used the old typewriter of my grandfather. It showed me the power of [[Uni-taskers]].

The typewriter can only write, not like a laptop where you can surf the internet, or play games or get distracted by social media. Just typing ahead. So refreshing.

That’s where I went down the rabbit hole of [[Distract-Free Typewriter|Distraction-Free Typewriter]] and bought myself a small Micro Journal. A digital device solely for typing. Obviously it can connect to the internet and could do much more - as I installed Linux on it - but the resources are so limited that already running [[Neovim]] that I remodeled to a Wordprocessor struggles to open. So there’s no danger of doing anything else.

Also because of the limited screen real-estate, it lets you really focus on the writing, and less so editing an article. So it’s really a joy to use to exercise my [[Creative Writing]] vein. Just for the joy of writing.

The Future-Proof Format: Markdown

As mentioned already, [[Markdown]] is the format of choice for me. Especially after I was trapped in Microsoft OneNote and its proprietary format, I couldn’t get my own notes out of it. It was super key to have something that will surpass the test of time. And there’s nothing more than [[Plain Text Files]] with Markdown.

I gave a full talk about this topic at Knowledge Management in the Digital Age: From Zettelkasten to Startup Owner, check that out if you want to know more why Markdown, its advantages over [[Rich Text]], and how you can build a note taking setup that works with Obsidian - and even setting the foundation for a solo business like mine.

Markdown has many more advantages. The format has been proven to be the best for agents. I can easily share my public second brain in Markdown with [[Quartz - Publish Obsidian Vault|Quartz]] with no conversion or manually copying notes between rich-text editors and website.

Markdown has all the advantages of simply writing with minimal formatting sugar. It has the advantage that the formatting lives as part of the text, which makes copy pasting not lose all the links or bold/italic etc, which we put for a reason.

Markdown is declarative, meaning you can automate things, you can have the same text, but different engines to present. E.g. I use [[HackMD]] for collaboration (Google Docs for Markdown), and I use Markdown to publish on my website. It’s the same file, the same format, there is no conversion needed.

Compare this to your typical Google Docs, WordPress, Webflow, or other [[Open Subscription Platforms]] such as Substack and Medium. These tend to enforce constraints, you need to always copy your text back and forth, creating copies of your text, potentially losing important formats.

The other big advantage, Markdown files are just [[Plain Text Files|Plaintext Files]], meaning we own the files - no big tech or company can forbid access for us or take them away, they work offline when we don’t have internet, and they are super fast as it’s just tiny files that are locally stored, no round trips to the server.

[!info] My Note-Taking Path from forgetting everything to Obsidian with Vim and Quartz
My path so far with note-taking:

1. Forgetting everything
2. Taking scattered and very detailed notes on multiple devices, apps, and paper
3. Improving during my studies with OneNote, where notes related to work or study go into separate notebooks (no notes for personal notes yet).
4. Starting to create a personal notebook for travels, personal research, etc.
5. There is still a lot of confusion about:
   1. where to store my notes
   2. changing of the folder structure
   3. finding older notes is complex and rarely happened
6. Switching to [[Obsidian]] with a new open format and a different spirit and capabilities.
7. Starting my [[Second Brain]]
   1. Constantly updating my long-time wealth of personal knowledge by adding notes about my health, journals, cooking, books I read, and everything related to my life.
   2. I Started to connect notes and sophisticate my system in a way that I confidentially find it later down my life span, the moment I need it.
8. Start using [[Vim]] and, more importantly, its [[Vim Language (and Motions)|motions]] for fast and effortless note-taking.
9. Sharing them publicly with [[Quartz - Publish Obsidian Vault|Quartz]].

Find the full break down on My Obsidian Note-Taking Workflow.

Wrapping up

I wanted to write much more about the art of writing, how to [[Writing|write]], [[How to Write Well]], and generally more the technique and art of writing. But as this article is already very long, I will save that for another long, distilled blog post in the future. You can follow the above links already, where I wrote a lot, but not in this distilled blog format.

Maybe one day I will write a book about it, I have so much more to say and tell 🙂. What do you think, would you read it? 🙃

Now I want to leave you with some unexpected impacts that writing had on me, and how to get started with blogging:

• My articles and website focus on data engineering, but my most successful posts (in views and virality) are topics about [[Obsidian]], [[Vim Motions for Writers|vim]], and philosophy
• That was a surprise — but now I get it: these were where I had something on my heart, something I poured many years into and put into a single writing
• Career impact: got higher salaries because I was considered known “world-wide” through my blog
• People knew my writing, and typically tend to like you as you give the writing for free
• When I started my own company, I basically didn’t have to sell — being online for so long, people know my writing, my principles, and even my life through my [[Second Brain]]
• The total set of articles is what matters: when people come back to you, that’s the [[compounding]] effect

[!question] The Elephant in the Room: AI
You might ask, but what about AI content, isn’t that a reason to not start to write? I’d say no for the same reason writing was good before AI and before books, and before time.
It’s good for yourself, to calm down, let out all your thoughts. Distill, learn and also remember things that are very important to you.

[[Writing Manually]], as I like to call it, is also my joy of writing - if you enjoy it, I enjoy it. It’s hard, it’s a challenge, it’s not easy. I get great satisfaction. Like chess, computers are much better, but we still play chess. And also for the love of words and communication, writing is communication. And to prevent more “AI Slop” from being created.

Read much more on [[Will AI replace Humans]], where I share my latest on that topic.

I have so much more to say, which a lot of it is in my second brain, so feel free just to browse and explore my second brain. Use the backlinks and the graph to explore more. I even added a semantic search, so you can find hidden connections on my public brain, on topics that might interest you, or on this very topic you just read.

A good book recommendation that is related (I share more on Book Recommendations and Notes), that helped me a ton, which I also wrote about finding mine, is The Pathless Path by Paul Millerd. This is the Tim Ferriss 4-Hour Workweek book, updated for today, and it inspired me to take the step of going full-time as a freelance technical writer and making writing my job.

What if you want to get started yourself? Read my interview on “Write that Blog” where I share more suggestions about starting your own blog. But generally, [[writing is hard]], just get started.

Have a note taking app (one only, based on an open format, available offline too), and write things down. Save important moments in your life, blessings of people telling you, insights from books you read, etc.

Follow the mantra of [[Learn in Public|Learning in Public]]. Use the feedback loop, let’s share and learn together. And know that over time, all your notes and personal knowledge will compound, like money does if you invest it cleverly.

[!note] Maybe the easiest way to get started: just an email converted to a blog

Hey World has this feature integrated into their email — you can write an email as you would normally, just a different recipient, and if sent, it will be online as a normal blog post. A very easy way to get started.

I’m leaving some links to tips and tricks for unblocking and finding a good rhythm.

• [[Coffee Break Rhythm]]: Using location pressure as a productivity forcing function. For me, in summer, it’s moving every 1.5-2 hours from coffee shop to coffee shop, creating a pressure to finish up before leaving. It tricks the brain into not thinking: “Oh, I have still all day long time” and then procrastinate, the opposite of [[Productive Procrastination]], where we let go on purpose to get some insights we wouldn’t have gotten otherwise.
• Use the doubts, the signs that you can’t write today. Take a walk instead, best would be to walk everyday. I call it the [[Productive Procrastination]]. Many see it as a bad thing, but it’s unavoidable, and me, and also others, believe it’s our body, gut telling us something. E.g. Tim Urban from Wait But Why is also big on procrastination. He says the same. He hates to procrastinate sometimes, but that’s how his brain works, and where he gets some insights he wouldn’t have gotten without.
• [[Ultradian Rhythm]]: Know that the first 10 minutes of a 90-minute deep work block are always going to be hard. That’s okay. Remind yourself of this when starting is hard.
• Embrace the New Luxury of Boredom: I feel we are at a turning point. We, the people, want to own things, want distraction-free experiences, and above all, want tools that benefit us, not the pockets of large companies. There are more stories that people use old iPods for music, buying the music. Typing on a typewriter solely for writing (like I did distraction-free typewriter), leaving the cloud, or just using local first products like Obsidian, DuckDB. Devices that are uni-taskers, doing one thing well.

Now it’s time to see what Git-like tools for data are out there, and how they actually work in practice. Part 2 dives into the tools and implementations. We’ll examine LakeFS, Dolt, Nessie, MotherDuck, Bauplan, and more, exploring how they work under the hood. Each tool takes a different approach to the same fundamental challenge: enabling Git-like workflows without copying petabytes of data.

The key insight from Part 1 was that all these tools separate metadata from data, using techniques like copy-on-write and pointer manipulation. But the devil is in the details. Some tools version entire data lakes, others focus on databases. Some support full merge workflows, others prioritize instant forking. Understanding these trade-offs will help you choose the right solution for your stack.

There will be gaps, and implementations are changing fast, so take it with a grain of salt. But this should give you a good overview of what’s out there, and help you invest more time in the ones that fit your use case best.

Let’s get into it.

Git-like Tools: Overview

There are many tools out there, some of which have been used for years, and others are rather new. We compare them and see what each of them has to offer.

Comparison Overview

The overview below serves as a summary. We will go into more detail, with each tool getting one short chapter with a showcase of features and application use cases.

It’s by no means complete, but it shows the most dominant players.

Further analysis of the OSS ecosystem of git for data tools and their GitHub activity tells us how healthy the repos are, as of February 2026:

And community responsiveness based on ossinsight.io, latest available month - click on link below to get a deeper insight in each repository:

Note: All data from GitHub API, Feb 2026. Github Activity Chart. See also GitHub Star History

Dolt stands out with the fastest PR merge times (~30 min median). lakeFS leads in total PR creators (178), reflecting a broad contributor base. Nessie’s near-instant issue response reflects automated triage.

[!note] How Do They Work?

While Git versions code through file snapshots and diffs, data tools must handle actual data, if possible, without copying entire datasets. Each tool solves this challenge differently, but they share a common approach: separating metadata from data.

Instead of duplicating data, they track pointers and references, enabling instant branching/cloning and zero-copy operations.

What usually happens without tools like this Testing in Production

Find more insight about the architecture and behind the scenes in Part 1, Branch, Test, Deploy: A Git-Inspired Approach for Data.

Git-like Tools: Break down

Let’s get started with the tools and see their features and how they work, categorized into three categories: data lake based, transactional and relational databases, and analytical databases.

Data Lake Versioning (Object Storage)

Data lake versioned tools sit between the compute engine and the object storage (S3, GCS, Azure Blob), leaving you free to query with whatever engine you prefer: Trino, Spark, DuckDB, etc.

LakeFS

LakeFS is one of the first tools to bring git-like versioning to object-storage-based data lakes. Its core approach is a metadata layer over object storage with immutable data and logical-to-physical address mapping on top of an object store such as a data lake, hence “lake” as part of the name.

It segregates data data/ with random physical addresses from its metadata _lakefs/, which includes range files, meta-range files, and commit information.

When you upload allstar_games_stats.csv to branch main, lakeFS generates a random physical address like s3://bucket/data/gp0n1l7d77pn0cke6jjg/cg6p50nd77pn0cke6jk0. This ensures immutability and files are never overwritten.

LakeFS operates as an S3-compatible gateway, intercepting read/write operations and managing versioning transparently. Applications interact with it like normal object storage while getting full Git semantics underneath.

The system implements a layered architecture:

1. Graveler: Core versioning engine managing branches, commits, and merges
2. Storage Adapter: Interfaces with S3/GCS/Azure
3. Hooks: Pre-merge and post-commit validation

Creating a branch from the CLI is as simple as this:

1

lakectl branch create lakefs://quickstart/denmark-lakes --source lakefs://quickstart/main

The UI supports creating pull requests, or branches, literally like GitHub but for data.

Check out their GitHub repo, documentation, or a practical example of Implementing a Write-Audit-Publish (WAP) Pattern for much more information.

Nessie

Nessie came out of Dremio and is another early adopter that has been doing this for a long time. Its core approach is a transactional catalog with Git-like versioning for Apache Iceberg and Delta Lake tables.

Rather than versioning data files, Nessie versions the catalog metadata, the registry of tables and their locations.

This separation enables zero-copy branching where branches share table metadata pointers, multi-table transactions with atomic commits across multiple tables, and Git semantics such as branch, tag, merge, and cherry-pick operations.

Nessie leverages the immutability of modern table formats with Iceberg:

1. Iceberg snapshots are immutable: Each table change creates new metadata.
2. Nessie tracks which snapshot each branch points to.
3. Branching copies pointers, not data or metadata files.
4. Merging updates pointers to replay changes from source to target.

Example workflow:

1
2
3
4
5
6
7
8
9

# Create branch
catalog.create_branch('experiment', 'main')

# Modify table on experiment branch
spark.sql("INSERT INTO catalog.experiment.orders VALUES (...)")
# This creates new Iceberg snapshot, Nessie updates experiment pointer

# Main branch unchanged - still points to original snapshot
spark.sql("SELECT * FROM catalog.main.orders")  # Original data

Nessie runs as a REST service with pluggable backends including metadata storage such as PostgreSQL, DynamoDB, or RocksDB, data lake integration that works with any Iceberg-compatible engine (Spark, Trino, Dremio), and version control with a Git-like commit graph with branches and tags.

Nessie doesn’t touch your data files. It’s a lightweight coordination layer that brings Git semantics to your lakehouse by versioning the catalog. This makes it complementary to tools like lakeFS (which versions data) and ideal for multi-table transactional workflows. Read more on GitHub.

Bauplan

Similar to LakeFS, Bauplan calls itself the programmable data lake and is a code-native platform for versioned pipelines, built on Apache Iceberg and initially optimized for ML. It’s not open source. Bauplan is built on a Python-first serverless lakehouse and is rather new.

Bauplan treats your data lake as a Git repository where:

• Data branches are first-class citizens, not just pipeline configs.
• Every pipeline execution is a commit with full lineage.
• All tables use Apache Iceberg format (Delta Lake compatible).

Creating an isolated branch with new snapshots of Iceberg tables from the CLI is as simple as this:

1

client.create_branch('experiment')  # Instant, zero data copying

It supports merging verified using Alloy model checking:

1

client.merge_branch(source='experiment', target='main')

The way it works is that it integrates a commit’s changes into another branch and uses Alloy, a lightweight model checker, to stress-test the core logic behind merging (also used for checking branching and commits).

The merge operation tries to detect conflicts at the table level, performs three-way merges for compatible changes, and creates merge commits preserving lineage. Find more info on Git-for-Data Semantics: Safe Branching & Merging at Scale or their implementation of the WAP pattern.

Bauplan brings Git’s full semantic model with branch, merge, commit, and revert to lakehouse data while maintaining compatibility with standard Iceberg tables accessible from MotherDuck, Snowflake, Databricks, or Trino.

[!tip] Software Modeling with Alloy

I haven’t heard of Alloy before, but it’s used not to model data, but for software modeling. It’s used for a wide range of applications from finding holes in security mechanisms to designing telephone switching networks. And now for git for data with Bauplan.

[!note] New Whitepaper Out

After this article was written, Bauplan released a new whitepaper on Building a Correct-by-Design Lakehouse that researches around pipeline boundaries with Git-like data versioning for review and reproducibility, and transactional runs that guarantee pipeline-level atomicity.

Transactional and OLTP Databases

These are row-oriented, ACID-compliant databases where Git-like versioning applies mostly to application data where we need to keep user records, orders, and schemas.

Supabase, Neon and Dolt are interesting because these are not data lakes, not based on object storage, and not analytical databases, but relational databases.

Supabase

Supabase’s core approach is full instance branching. Each branch is a completely isolated Postgres database with the entire Supabase stack (Auth, Storage, Realtime, Edge Functions).

Supabase branches create separate environments that spin off from your main project, allowing you to test changes like new configurations, database schemas, or features without affecting production.

It works by creating a Git branch and opening a pull request. Supabase automatically launches a Preview Branch and runs migrations from the repository’s migrations directory. Each branch gets a dedicated Postgres instance with a unique connection string and APIs, isolating them from production and other branches.

Creating a branch via GitHub integration:

1
2
3
4

# Automatic with GitHub integration enabled
git checkout -b feature/new-reports
git push origin feature/new-reports
# Supabase automatically creates preview branch when PR is opened

Or via the CLI:

1

supabase branches create feature-branch --project-ref your-project

When merging, migrations in the repository’s migrations folder run incrementally on each commit, allowing you to verify schema changes on existing seed data. When you merge the PR, those migrations automatically apply to production.

As each branch is a new Postgres instance created from scratch, the approach is conceptually simple but requires branches to be seeded (manually populated with test data since production data isn’t copied) with data since they start empty. Each branch incurs its own compute and storage costs. Read more on Branching Supabase Docs.

Ideal for full-stack development where you need the entire backend stack (database + auth + storage + functions) to test features end-to-end.

Neon

Neon is a serverless Postgres platform (now part of Databricks) whose core approach is copy-on-write storage-level branching. Unlike Supabase which spins up a full new instance, Neon branches at the storage layer, making them instant regardless of database size and including the actual data.

Each branch is a new timeline in Neon’s custom storage engine. No data is physically copied. The branch simply starts from a pointer to the parent’s state at a specific LSN (log sequence number). Pages only diverge when writes happen, so you’re billed only for the delta.

1
2
3
4
5

# Create a branch from the CLI
neon branches create --name feature/user-auth

# Branch from a specific point in time
neon branches create --name recovery --parent 2025-01-15T10:00:00Z

Neon also supports snapshots (named, immutable point-in-time saves, like git tags) and rollback via finalize_restore: true, which restores a snapshot onto the active branch in-place while preserving the stable connection string. There’s no reconfiguration needed. For safe experimentation, finalize_restore: false creates a temporary preview branch instead.

The key limitation: Neon has no merge support. Branches diverge but can’t be reconciled automatically. Changes are applied back to production using standard migration tools.

Ideal for database-focused workflows where you want instant, full-data branches with production-like data out of the box, and don’t need the full backend stack.

Dolt: Git + MySQL

Dolt is a SQL database that you can fork, clone, branch, merge, push, and pull just like a Git repository. It’s a MySQL-compatible database and is fully open-source. Dolt’s core approach is a SQL database where every row is versioned, combining Git’s commit graph with MySQL’s query interface.

Dolt stores data in a content-addressed graph using Prolly Trees, a novel data structure that enables cell-level version history, efficient structural sharing between versions, and fast diffs and merges.

Every database operation can be committed with:

1
2

INSERT INTO employees VALUES (1, 'Alice', 50000);
SELECT DOLT_COMMIT('-am', 'Add Alice to payroll');

The commit creates a snapshot of the entire database state at that moment, stored in the commit graph just like Git. Unlike traditional databases, you can diff any two versions:

1
2
3
4
5

-- See what changed between commits
SELECT * FROM DOLT_DIFF('main', 'feature-branch', 'employees');

-- Show cell-level changes
SELECT * FROM DOLT_COMMIT_DIFF_employees WHERE from_commit='abc123' AND to_commit='def456';

This enables cell-level audit trails with diffs showing which rows were added/deleted/modified, which cells changed with their before/after values, and who made the change via commit metadata.

Dolt implements Git commands almost literally. You can run dolt with any of these commands: branch feature-123, checkout feature-123, add ., commit -m "Add new customers", push origin feature-123, checkout main, merge feature-123.

You can even push/pull to DoltHub (like GitHub for databases) or run Dolt as a MySQL replica for existing applications.

Dolt uses copy-on-write with structural sharing where unchanged rows are shared between branches via pointers, and modified rows create new leaf nodes in the Prolly Tree.

This means cloning isn’t “free” like with lakeFS, but it provides true database semantics with ACID transactions.

There’s much more. Read more on their GitHub.

[!note] Worth noting

DoltgreSQL, the Postgres-compatible version of Dolt, reached Beta in 2025 and is available on Hosted Dolt. If your stack is Postgres-based, DoltgreSQL brings the same Git-like versioning semantics without requiring a MySQL migration.

Analytical Databases & Warehouses

These tools are OLAP-style and analytical-style databases optimized for read-heavy analytical queries.

MotherDuck

MotherDuck, as a cloud data warehouse, implements versioning differently from dedicated Git-for-data tools, prioritizing operational convenience over full version control semantics. With the addition of named snapshots, it gets even closer to Git-like semantics.

It offers two types of snapshots. Automatic snapshots: Created continuously in the background (roughly every minute when no writes are active). These are governed by SNAPSHOT_RETENTION_DAYS. These are configurable up to 90 days on the Business plan, defaulting to 7 days. They provide point-in-time recovery without any manual intervention.

And named snapshots that you create explicitly with CREATE SNAPSHOT. These are not subject to garbage collection as they persist indefinitely, even if the source database is deleted. Think of them as Git tags for your database, a permanent bookmark of a known-good state you can always return to.

The git analogy maps well:

1. CREATE SNAPSHOT → git tag: bookmark a known-good state
2. CREATE DATABASE ... FROM → git checkout -b: isolated environment from a snapshot
3. ALTER DATABASE SET SNAPSHOT TO → git reset --hard: roll back to a previous state
4. UNDROP DATABASE → recovering a deleted branch

Combined with zero-copy cloning and database sharing, this enables practical git-like workflows. While MotherDuck doesn’t support Git-style merging, COPY FROM DATABASE (OVERWRITE) acts as a replace, somewhat like a merge without conflict resolution. Combined with snapshots and zero-copy clones, this gives you a practical branch-modify-promote workflow:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

-- 1. Snapshot production before changes (persists indefinitely)
CREATE SNAPSHOT 'pre_release_v2' OF production;

-- 2. Clone from that named snapshot to an isolated dev database (instant, zero-copy)
CREATE DATABASE dev_branch FROM production (SNAPSHOT_NAME 'pre_release_v2');
-- Or clone from a point in time: (SNAPSHOT_TIME '2026-01-28 08:00:00')

-- 3. Make and validate changes on dev_branch
-- ... run transforms, test queries ...

-- 4. Promote: overwrite production with dev_branch (instant, metadata-only)
COPY FROM DATABASE dev_branch (OVERWRITE) TO production;

-- 5. If something goes wrong, restore from snapshot
ALTER DATABASE production SET SNAPSHOT TO (SNAPSHOT_NAME 'pre_release_v2');

This operates purely at the metadata layer and is nearly instantaneous. It’s not a true merge (it’s a full replacement, not a diff-based reconciliation), but for many data workflows where you want to validate changes in isolation before promoting them, it covers the key use case.

[!example] Deep Dive

If you want to know even more about how to use named snapshots and generally rolling back to a certain time, this blog More Control, Less Hassle: Self-Serve Recovery with Point-in-Time Restore goes into more details.

DuckLake

DuckLake is the open lakehouse format that uses a SQL database as its metadata catalog instead of JSON/Avro manifest files. DuckLake is relatively new (with 1.0 around the corner and its first release in May 2025), so you could use other mature open table formats like Apache Iceberg, Delta Lake or Apache Hudi.

But DuckLake has its relevancy for git-like workflows because:

1. Snapshots are Git commits: Every DuckLake change creates a snapshot with author, commit message, and changeset tracking. This is the closest to actual Git semantics in the data lake world.
2. SQL-native metadata: Uses DuckDB/PostgreSQL/MySQL as catalog, so metadata operations are standard SQL transactions. No manifest file scanning or compaction storms like Iceberg.
3. Millions of snapshots: Snapshots are just a few rows in the catalog DB. No need to proactively prune snapshots (a major operational burden with Iceberg).
4. Time travel + change feed: Query any table at any version, track insertions/deletions between versions.

With MotherDuck (fully managed):

1
2
3
4
5

-- Fully managed DuckLake on MotherDuck
CREATE DATABASE my_lake (TYPE DUCKLAKE);

-- Or bring your own S3 bucket
CREATE DATABASE my_lake (TYPE DUCKLAKE, DATA_PATH 's3://my-bucket/lake/');

[!example] DuckLake Example

See valuable examples and DuckLake workflows in DuckLake workshop.

Related Data Engineering Git-like Workflows

Besides storage for data, which is the most important part and at the same time the hardest as we need to deal with state, it’s not the full picture. We have DataOps to handle the full picture.

Data pipelines and their code also need to be deployed on a clone or branch, so how do we do this? One example is orchestration.

Orchestration: Dagster Branch Deployments

If we look at the full picture of the data engineering lifecycle, we need more than just storing data in a git-like manner. To support the full lifecycle, it would be best to run everything in a git-like style to roll back or switch branches. It’s great to see that orchestrator tools like Dagster and others also have this functionality included.

Meaning branching does not only apply to the data, but also to data pipelines, and we can set a run automatically. Dagster is doing that with their cloud solution, integrating GitHub workflows with PRs and actions.

Dagster’s core approach is lightweight staging environments created automatically with every pull request that branch both code and data. Branch deployments deploy your branch on Dagster+ as a separate deployment. This only works if your underlying technology supports cloning. For example, as we’ve seen, one of the above tools that supports cloning will allow Dagster inside the deployment to clone relevant data into that new branch deployment.

On PR creation, it will automatically create a staging environment with a branch, launch jobs to configure the test environment including cloned data(base), and allow parameterized pipelines to test. If the tests pass, you can approve the PR, and it merges and automatically deploys to production with the right CI/CD pipeline.

Orchestrators and other data stack tools depend on cloning support and features such as branching for a true isolated environment. As Nick Schrock noted in the Data Engineering Podcast, this is similar to the challenge with Apache Spark where testing locally is nearly impossible. Branch deployments solve this by branching the entire environment.

This is extremely powerful as it replaces the need to copy data locally or set up complex staging environments. You get a true production-like test environment that’s automatically created and destroyed with your git workflow. Read more on Dagster Branch Deployments.

AI Agents: A Branch for Testing

Lastly, this also works well in the realm of AI agents that help us test based on a branch or snapshot. This is similar to git worktree for small git repos with code where basically each branch is a separate folder and we can work and change different branches simultaneously without breaking any of the other branches or data.

Once we have a working branch with data included in isolation, we can send off an agent autonomously, and let it open a PR to review. This way we have a clear gateway before it goes to production, we can test it on that branch, including its data, and merge when all looks good.

Based on its own fork, we can avoid collisions, instantly roll back or delete a branch and start again, have perfect consistency as data is frozen and locked for the agent to work on, and clean debugging as no other ETL data pipelines interfere.

Conclusion

So where does this leave us? In Part 1, we established that Git for data is fundamentally harder than versioning code because we’re managing state at massive scale. We learned about the efficiency spectrum, from metadata pointers to full copies, and why zero-copy operations matter.

Now, having explored the actual tools and their approaches to git-like workflows (LakeFS, Dolt, Nessie, MotherDuck, and others in production today), we know a little more about how it all works. Each tool makes different trade-offs, but they all solve the same core problem: how do you version data without copying petabytes.

The answer, to me: separate metadata from data. Whether it’s LakeFS’s random physical addresses, Dolt’s Prolly Trees, Nessie’s catalog pointers, MotherDuck’s zero-copy clones, or Neon’s branching feature, they all use clever tricks to make branching instant. Some focus on data lakes, others on databases. Some support full merge workflows, others prioritize instant forking. Your choice depends on your stack:

• LakeFS and Nessie excel at data lake branching with zero-copy efficiency
• Dolt brings true Git semantics to SQL databases
• MotherDuck offers named snapshots and zero-copy clones for cloud data warehousing, with DuckLake adding SQL-native time travel
• Bauplan focuses on versioned pipelines and ML experiment reproducibility
• Neon and Supabase provide branch/fork-based workflows for isolated testing

The ecosystem is still evolving. Maturity varies across tools, with different workarounds to limitations that best fit data in a git-like workflow. Some trade merge capabilities for instant forking. Others require infrastructure changes. The key is picking what fits your workflow and scale.

Start small. You don’t need to instrument your entire stack overnight. Look at your recent production incidents: which pipelines caused them? Those are your highest-risk areas. Add branching there first. Test changes on prod-like data before deploying. Build confidence through small wins, then expand.

We want to bring the same confidence we have with code versioning to the stateful world of data. And with tools like Dagster’s branch deployments and emerging AI agent workflows, we’re seeing Git-like patterns extend beyond just data storage into the full data engineering lifecycle.

Git-like workflows are becoming table stakes. Maybe not today or tomorrow, but with the right tools and changes in workflow we can achieve significantly better change management, testing on production data, fast rollbacks, isolated experiments, and most importantly, peace of mind when deploying changes.

That’s the promise. What’s your experience? Have you tried it? Do you run any of the above in production? I’m curious to hear more.

Appendix

While I was writing this article back in November 2025, Tigris was an interesting database contender with Supabase-like features such as forked buckets and zero clone. But at the time of this publishing, the GitHub repo got archived, and therefore removed from the comparison in this article.

Full article published at MotherDuck.com - written as part of my services

So I built one. A RAG system that runs locally with DuckDB as a vector database, then syncs to MotherDuck for a serverless web app running entirely in the browser via WASM. Think of it like J.A.R.V.I.S¹ for your markdown files: search about a topic, and it shows connected notes up to two hops away, semantically similar content, and hidden connections between ideas that share no direct links.

In this article, I walk through how I built this and how it works, from using DuckDB’s vector extension locally to serving embeddings through MotherDuck’s WASM client. Along the way, you’ll see how data engineering skills can make use of lots of note-markdown files. If you want to dive straight into the code, it’s all on GitHub at Obsidian-note-taking-assistant, and you can try the web app on my public notes at Explore RAG.

For building the web app I used Claude Code and it came together in a few hours using the plan mode. This approach is powerful for any data engineer building pipelines or related work, especially when you have a clear vision of what you want. The big productivity boost wasn’t only the model getting smarter, in my opinion, but something else, more on that in the article.

This is how it looks. Let’s talk about how I built it and some behind the scenes.

Vision & Why I Built This

I have 8963 local notes (according to find . -type f -name '*.md' | wc -l) in my Obsidian vault, some are very long, and there are more images and PDFs connected. Wouldn’t it be nice to have an insight from my own thinking a while back, or some quotes I forgot², or things you didn’t think of?

The requirements that I set myself were to use Obsidian backlinks as these are already curated and well structured as a graph-like organization. I wanted to see notes that are multiple hops away and hard to see without a tool. I wanted to search non-obvious neighbors or similarities and also show me hidden connections that would be interesting, both locally and online. These are especially helpful in the brainstorming and initial phase when starting an article or a note, giving me new ideas on existing notes I have written once in my life.

Examples could look like this:

Show me my notes on Functional Data Engineering that relate to my current article (one or two hops)

Notes that are relevant from my vault. Or related ideas

Highlight any disagreements between the notes

Give me all notes I took on these matters and related, and give me the source note from my Obsidian vault

Such a tool is especially helpful during brainstorming when writing my articles, or when I journal some ideas or when solving a hard problem. All of this should be local, but also available as a web app, so I can share it with you and connect it to my public second brain.

Starting Position

With Obsidian, there are many Obsidian plugins such as Graph Analysis, Obsidian Smart Connections and many more, that let you do similar things. But some require to hook up a public AI provider, don’t work very well anymore, or don’t do exactly what I wanted.

The easiest would be to use Claude Code or any other agents, as it’s just Markdown files, but again, then you give away all your sensitive, potentially insightful notes and thoughts. That’s why I wanted to build an Obsidian knowledge assistant that is trained based on my data. I started with a simple Retrieval-Augmented Generation (RAG) system that uses DuckDB for storing vectors. I used Vector Similarity Search Extension for storing vectors and did a couple of tests with Claude Code.

I shared it online and got helpful feedback to use a specific model bge-m3 and integrated it as much as possible with the help of agents. I added the above requirements that it should use Obsidian native links and train based on my vault.

This was my first round. Building a job that creates chunks and ingests them into DuckDB with the vector extension Vector Similarity Search Extension.

I used two different modes, as the above takes more time to generate embeddings. I could run the BGE-M3 overnight and it was done after ~2 hours, not on all my notes, but on my public notes, which are 584.

Local-First

I started with the local-first approach because I want to be independent, and also I have sensitive or valuable notes that I don’t just want to give away or upload to the cloud.

But there are also other reasons why you might want to use a local model. Some say:

A.I. research done by a cloud service will hallucinate because you have no control over the weights or limits of the LLM. This is why anyone who wants to do A.I. should run their projects locally including Deep Research. Bsky

Additionally, a local model with lots of your own context to research with will be better suited for your use case. It doesn’t mean that it does not hallucinate, but what I find most useful is that suggestions and ideas are based on my own notes, which I sometimes have forgotten, or if new ideas, they are combined based on my research.

Web App

I added a web app that uploads the generated embeddings to MotherDuck and uses DuckDB WASM to serve in the client (web browser), so I could share the findings easily with anyone interested in my second brain notes.

This went really well, and I share all the details at the end of this article, with some lessons learned and how you can do it for yourself too.

Knowledge Assistant: Building a RAG for Data Engineers

Now let’s get to the building part. As initially explained, this article converts data engineering knowledge into a searchable tool. Hopefully finding new insights, related topics, and learning something new.

This is now done on top of my public (mostly) data engineering notes, but we might add code snippets, interesting quotes, etc. To me, all of these might just be text files, and mostly markdown, that’s why this system based on text files is so powerful. We can use it as context to help us more.

The outcome and connected web app looks like this:

What We Built: Retrieval Without the LLM

A Retrieval-Augmented Generation (RAG) system that is trained on our notes that we have (we use Markdown). More specifically: Obsidian Markdown, that has the advantage of links and backlinks that give us additional clues we can use.

RAG in particular is a technique that can provide more accurate results to queries than a generative large language model on its own because RAG uses knowledge external to data already contained in the Large Language Models (LLMs).

So what we built is only the Retrieval and Augmented part. We don’t use an LLM yet, only retrieval of relevant and hidden notes based on a search. Specifically notes, code snippets as parts of notes, and other relevant ideas.

Architecture with Embed Model, MotherDuck and Next.js

First I had to split my notes into separate chunks and connect relevant links.
This is done through an embedding model that converts text into numerical vectors, so we can compare meaning rather than just keywords.

This runs locally and two models can be used: all-MiniLM-L6-v2 (384 dimensions, fast for testing) and BAAI/bge-m3 (1024 dimensions, production quality). This is the top-level Python code in the GitHub repo. It provides a CLI and DuckDB database where we can search semantically, discover hidden notes, or traverse connected notes up to two hops away.

The chunking is markdown-aware: it respects heading boundaries, preserves code blocks intact, and splits on paragraph breaks. Each chunk stays around 512 characters and carries its heading context along. Before embedding, I prepend the note title and section heading to each chunk (e.g., "Title: DuckDB | Section: Installation | actual content...").

This acts as a semantic anchor and noticeably improves retrieval quality.

Disclaimer: I don’t have deep expertise in building RAG systems and semantic search, so this is built on the best of my knowledge and what helps me most in my daily work.

The ingestion pipeline creates these tables with relevant information:

• notes: Note metadata, content, frontmatter
• links: Wikilink graph edges
• chunks: Chunked content for RAG retrieval
• embeddings: 1024-dim vectors (BAAI/bge-m3)
• hyperedges: Multiway relations (tags, folders)
• hyperedge_members: Note membership in hyperedges

The second part is a web app served via a Next.js UI and a MotherDuck WASM client that connects directly to the MotherDuck cloud database from the browser.

This means no database server to set up or maintain. I added a FastAPI service on Railway to serve the BGE-M3 embedding model, which avoids API costs from Hugging Face (and also makes it reliable, since Hugging Face’s inference API kept timing out with the BGE-M3 model).

The architecture uses mostly serverless components:

Semantic search matches meaning, not keywords. When I search for “how to model data in a warehouse,” I want notes about dimensional modeling or dbt transformations to show up, even if they never use those exact words.

The BGE-M3 model converts each chunk into a 1024-dimensional vector, and we rank results by cosine similarity between the query and stored embeddings. Locally, DuckDB’s VSS extension handles this with an HNSW index.

In the web app, MotherDuck’s WASM client doesn’t have VSS, so I compute cosine similarity manually with DuckDB’s list functions. I was surprised how well DuckDB handles this without a dedicated vector database, one file for relational data and vectors together.

The “graph-boosted search” mode multiplies similarity by 1.2x for notes that are also graph-connected. Simple, but it surfaces better results because your link structure encodes intent that embeddings alone miss.

And the hidden connections feature, finding semantically close notes with no direct wikilink, turned out to be the most useful discovery tool.

It found links between notes I’d written months apart and never thought to connect.

Running It on Your Own Vault

As we constantly add and improve our “second brain”, this is very powerful, so we can just rerun the ingestion and we get the update.

This is built on my data, but you can use the provided GitHub repo and run the local make ingest job to run it on your own Obsidian vault or Markdown files. You’ll get the same UI and CLI to ask questions about your notes out of the box.

The results are tailored to our interests, needs, and even notes, as we are the ones who wrote the notes down. Or if you took a lot of highlights via web clippers ReadWise read-it-later, Obsidian Webclipper, also from other authors, but still snippets that you chose to store.

To run it on your own notes, clone the repo, set VAULT_PATH in the .env file to your Obsidian vault (or any folder of Markdown files), and run make ingest.

The ingestion parses all .md files, chunks them, generates embeddings with the BGE-M3 model, and stores everything in a local DuckDB file. From there you have the full CLI with semantic search, backlinks, connections, and hidden link discovery.

If you want the web UI too, sync to MotherDuck with make sync-motherduck and deploy the Next.js app.

The Final Result

The result of this exercise is two parts with sub-components like this:

• Ingestion pipeline: A local job that parses Obsidian markdown, chunks it, and generates embeddings using the BGE-M3 model. Run make ingest and the local DuckDB file is ready to query.
• Web app at explore.ssp.sh, composed of three services:
  • Frontend on Vercel: Next.js app with MotherDuck WASM client running DuckDB queries directly in the browser.
  • Database on MotherDuck: Cloud-hosted DuckDB, synced from local via make sync-motherduck. No server to manage.
  • Embedding microservice on Railway: A FastAPI endpoint that hosts the BGE-M3 model and converts search queries into vectors on demand. The browser sends your search text, gets back a 1024-dim embedding, and uses it to query MotherDuck for similar chunks. This avoids running a ~1.8GB model in the browser and sidesteps Hugging Face API rate limits.

Here you can see backlinks and hops that go over two notes. The hops are interesting as we don’t see this easily on a graph, or it’s harder to showcase. That’s why I added them besides the normal backlinks and outgoing links.

Find hidden connections. Here we see that AT Protocol, the protocol behind social media platform Bluesky and others, is connected to Ducklake. Something I wouldn’t have associated myself:

Now we can compare notes, think why this could be, and what’s the connection and insight we can gain from it. This is exactly why I built this, to get such insights.

[!info] Clickable Links

Each note on explore.ssp.sh has a clickable link to my public brain at ssp.sh/brain/[note-name].

Lessons Learned: AI Agents for Data Engineers

As you probably have noticed, since the Christmas break, the AI hype or enthusiasm around agents got very loud. One reason is that many got a good amount of time to actually test the latest. On the other hand, the models got better, and thirdly these AI companies provided new features such as Skills, cowork, and many more.

I myself also took some time and thought about how we can leverage agents for data engineering, especially Claude Code. But contradicting many who say the models got much better, I think the key to the boost of productivity is a different one. With nao, ChatGPT, Claude, and probably others, we have had AI agents and models already for a while, but most powerful at the current moment are the agents in plan mode. It’s the key to build longer and have us more in the loop.

But what is “Plan Mode” you might ask? The definition:

Claude Plan Mode is a read-only state in Claude Code, an AI coding assistant, that lets it analyze a codebase, ask clarifying questions, and generate detailed implementation plans without making any actual file changes or executing commands, ensuring safety and structure before development begins. It’s activated by cycling modes (often Shift+Tab) and is great for exploring, planning complex changes, and building context, allowing developers to approve the AI’s strategy before actual coding starts. More on What Actually Is Claude Code’s Plan Mode?

With that, it’s amazing what you can build. All the open todos we add to our backlog, we can now quickly build and test or solve, and think through the problem by actually laying out the step-by-step instructions. After it’s built we get a feel for it quickly and can give better feedback on whatever job we have at hand right now.

Still we need to be careful to not just jump into building every little thing, as we could, because spending hours on something that we don’t need is still wasting precious time.

I have experienced it myself often. I get the perception of being super productive, but after a couple of hours, or sometimes days, we actually didn’t achieve what we needed. The idea we thought was cool didn’t go anywhere, and we are mentally more exhausted because we didn’t really do the heavy lifting, meaning we don’t really understand what was generated. And potentially also didn’t learn anything new.

With that in mind, we need to be careful when to use the new tools, certainly not always, but there are many ways. So how else should we use agents and AI as data engineers and knowledge workers?

[!note] Plan Mode Support

Besides Claude, Plan Mode is widely adopted across AI coding assistants including Cursor (October 2025), Windsurf (June 2025), GitHub Copilot (VS Code, Visual Studio, JetBrains, Eclipse, Xcode), Lovable, Bolt.new, and Replit (September 2025). Everyone is following a similar pattern of letting AI analyze, ask clarifying questions, and propose structured implementation plans before writing any code.

Plan Mode: And How We Work Best with AI Agents

This is how we humans work best as well. We make a plan, and then execute it and adjust along the way. But it’s also a great way to work with juniors, and in that sense, AI agents.

Because we say what we want in an abstract manner, the agent says what it would do in a plan form (just a markdown file, markdown runs the world these days), and then we as the senior, or the designer or architect can see if it missed our interpretation (as language is not precise), and we work on a great plan with all the details. This way we know it does what we expect it to do. And then it goes off and does it autonomously with access to the terminal and all command line tools.

But there’s one more factor, it’s the human factor. Whatever it builds, it builds on trained data. So it will use what most people use. Which might be ok for most cases, but maybe not if you want to build something unique, innovative. That’s why I think for most writers, it’s not the right tool to let it write the stuff for us. Just for that fact, but even more so, the character and soul of the person gets stripped away. The quirky things someone does, which make them who they are, that takes away from the fun of writing.

Obviously in coding, this is not the same. Except if you are another programmer and need to read the code, no? Because any data engineer would love to read the code from a human rather than an AI, it’s kind of boring. But maybe it just needs to do the job, and not all human code is beautiful too, right?

[!note] See the Prompt for the Web App

If you want to know how I built the web app without having experience in Next.js, I am sharing the initial prompt with plan mode that could be interesting. The summary of the full session (ca. 3-4 hours) is at build-summary.md.

Where Are We Heading?

So what about data engineering? Where are we today?

As I have written extensively about at Self-serve BI thanks to AI or using it for data modeling along with semantics, speed, and stewardship, humans still need to be in the loop, and we need to be careful to not generate too much (ingestion logic, business logic, general code, or dashboards) that is unmaintainable or never needed in the first place.

On the other hand, there’s no definitive answer right now, we are all just figuring it out. That’s why some say it’s the most exciting times, because everything is supposedly going to change. Andrej Karpathy said:

Clearly some powerful alien tool was handed around except it comes with no manual and everyone has to figure out how to hold it and operate it, while the resulting magnitude 9 earthquake is rocking the profession.

As a writer but also data engineer, I find it most useful when it suggests notes and ideas I have forgotten about that are relevant to my current task at hand. Or a snippet of code.

Repeating Code Snippets over and over

How many times have we written an ingestion pipeline that does the same thing just for a different source? Written an incremental update pipeline, or a full load, or implemented Slowly Changing Dimensions (Type 2).

Wouldn’t it be great to have a tool that helps us remember and suggest code that worked for a problem at hand? No wonder Windows has a built-in Windows Recall feature that takes snapshots of everything we do, so we can see and remember what we did. Google traces where we went on Google Maps Timeline, and so on. Not saying all of these are good, but clearly there’s a need for it.

Vibe Coding

Mostly these tasks are called vibe coding these days. I believe that vibe coding is best when you have an existing framework present and it can extend it. E.g. your website skeleton that already has a pre-existing structure is much better than starting from scratch, especially maintainability-wise.

Also, the more it has to predict in the future, the more likely it will introduce errors, compared to you providing a big skeleton with all the needed files and just extending on functionality.

This is the same for data engineering too. Declarative Data Stack, YAML Engineer is exactly that. A well-designed YAML that has a powerful system in the backend can go a long way with an agentic and vibe-coded approach.

It’s similar to Spec Driven Development (SDD), which is when we write our instructions in claude.md and Claude or any AI agents implement this. Also what Esco Obong said about what they do at Airbnb: the hard part is coming up with the spec, talking to business, etc. The coding part is the small part.

And this is also where the human is still dearly needed in my opinion. Human in the seat and config-driven development is what it comes down to with AI agents. Plus, AI models have a context limit. Sure, we humans do too, but we can think more across domains and understand intuitive things that might not work for a statistical model.

This shows how that works, and why Markdown is in the middle of everything. Not only for the knowledge, but also to build and develop things.

[!tip] Lifehack for Prompting

Always keep it simple, because it’s easy to make it complex. The true beauty lies in making it simple, which is something agents are not good at.

Use MCP

A key was using MotherDuck MCP with a direct connection from Claude Code to the database while prompting the initial version. Claude could directly query the database and its columns to implement the actual web app (see the initial prompt here).

Meaning Claude (in my case) could just query the database, use SHOW TABLES, select them, and extract their data types. And more, learning about the content and graph relationships that I had built in the first part.

So Claude could easily build a first version based on my instructions and existing DuckDB database. I also shared the great docs to build Customer-Facing Analytics Guide in a (3-tier Architecture).

With that, I almost had my web app ready with a single plan mode prompt.

[!example] Claude supports LSP now

As code editors do, Claude also supports LSP (Language Server Protocol). This helps speed up Claude to read the code more efficiently, doing lookups by jumping to references or definitions instead of searching its way through the code. It might also understand the code better as it has a language server to use.

Conclusion

Building this tool reminded me again how powerful DuckDB and MotherDuck are. It’s a Swiss Army knife database that can handle unique tasks and simplify my note-taking by providing a serverless database for querying my embeddings.

Now I have a powerful tool to search for related notes when I need to solve a problem, or to find relevant notes in my own second brain. The hidden connections this tool surfaces are valuable only because they’re my connections, my thinking, not just crawled information on the internet. And not only that, I can even provide a minimal but useful web app for you to search my public notes, too.

As for the AI agents that helped build it: they got me there faster, but only because I stayed in the loop. Let them run without direction, and you’ll get a thousand lines solving the wrong problem. To me, the “human” architect is still needed.

Other implementations I have collected over the years or came across while building this that might be helpful if you want to build something similar.

If you have many more files and embeddings that need to be created, follow the Using DuckDB for Embeddings and Vector Search article that runs on the GPU, creating embeddings for 2.85M Wikipedia articles. He used the Arrow/GPU acceleration and batch inserts via Arrow.

Some more links and repos I found interesting:

• Scalable Embeddings & Vector Search
  • Using DuckDB for Embeddings and Vector Search: Tutorial on GPU-accelerated vector search that created embeddings for 2.85M Wikipedia articles using Arrow batch inserts and HNSW indexing.
• Local-First Search Tools for Markdown
  • qmd: Tobias Lütke’s CLI search engine combining BM25, vector search, and LLM re-ranking—all local via Ollama, works with plain markdown (no wikilinks needed).
• Obsidian AI Assistants
  • Obsidian Copilot: A popular Obsidian AI plugin (6.1k+ stars) with vault chat, agent mode, and image/PDF/web processing—no index required for basic search.
  • Chat with Your ENTIRE Obsidian Vault OFFLINE (YouTube): Video walkthrough of offline Obsidian vault chat with Claude 3 integration.
• RAG Frameworks & Libraries
  • Quivr: YC-backed opinionated RAG framework (38.6k+ stars) supporting any LLM, any vectorstore, and any file type with YAML-configured workflows.
  • LennyHub RAG: Complete RAG implementation on 297 podcast transcripts with knowledge graph extraction, Qdrant storage, and interactive network visualization.
• AI-Assisted Development in Production
  • Esco Obong on AI Coding at Airbnb (LinkedIn): Airbnb engineer shares writing 99% of code with LLMs, noting that code is “only a small part of the actual work.”
• My List of Obsidian Related RAGs: Second Brain Assistant with Obsidian

Full article published at MotherDuck.com - written as part of my services

Back then, I had a checklist of basics I needed before I could commit to Linux as a daily driver: Obsidian, a Raycast-like launcher for fuzzy finding files and folders, screenshots (Snagit), daylight adjustment (f.lux), calendar events in the top bar. Those were quick wins.

Eight months later, I’ve gone through many more challenges and learnings. In this post, I’ll share which apps replaced my heavily integrated macOS workflow, what my Omarchy workflow looks like now, and — honestly — what still doesn’t quite work.

Apps that Replaced My macOS Apps on Linux

Let’s start with which apps and how I changed some of my workflow now in Linux.

Below list goes from complex Raycast replacement that was integrated into my whole workflow with search through files, calculator, emojis to calendar, daylight gamma correction for night sessions to PDF viewer that replaces Finder to sharing screen with Linux window picker, and much more.

It continues with running Windows on Linux with a simple install toggle and finding the right hardware, before I create a conclusion of these initial months using Linux full time for my business and also privately.

App Launcher and Raycast Replacement: Fuzzy Search, File Search, Clipboard, Math, and so on

One of the first apps to replace that most have, and that I also used, is Raycast. It’s an app I couldn’t live without, not only for the fuzzy finder but also for quick calculations, searching files, and clipboard manager.

With [[Walker Launcher]] I found the perfect replacement which has this all included and works like a charm.

Search file content with spotlight - Find files with Walker with built-in preview: ![[img_Switched from macOS to Linux- 6 months in_1770740249453.webp]]

Opening its containing folder or file with the default program. This is how I search and find anything compared to manually browsing through file explorer. Find any files within seconds with built-in search of Walker (Before I found Walker, I built my own one).

Emojis quick search. It comes with Walker built-in too, but I have my own script so I can find emojis faster as I can change the search term. Very simple, but powerful
![[img_Switched from macOS to Linux- 6 months in_1770741956438.webp]]

Clipboard managers, of which there are several, but Walker comes with one built-in too. Including search and image preview:

Other dedicated clipboard managers are cliphist or Clipse. There are also other Raycast-compatible launchers for Linux such as flare, Rofi, and many more.

Keyboard Shortcuts and Quick Symbols

I used Kanata for integration of advanced features to switch between my keyboards and some of the advanced use cases such as using CAPS LOCK for vim-like movements. I use caps + hjkl to move left, down, up and right with the respective arrow keys as almost all programs work with arrow keys. Also F1-F12 functions with caps+1 for F1.

For simple replacements, I used XCompose to write Umlauts (äöü and special symbols —«» and more). I used Karabiner-Elements heavily, and Kanata solved it for me, see my configs at dotfiles.ssp.sh/kanata.

Backups and Data Sync

Time machine on macOS was great. I used sync.com for dropbox-like sync on macOS too. Neither worked on Linux. So I switched to Filen, which has a similar setup and stores the data encrypted, and hosted in Germany. I’m using Stow for all my dotfiles stored in Git. It’s great, check them at dotfiles.ssp.sh.

I back up my images, personal documents, or scripts also with rsync-scripts to save to my homeserver and encrypted drive on Vultr. See more Tech Independence.

I also looked at NextCloud for hosting it myself, but for now I just need something that works. As Filen is an Electron app, it just works everywhere.

Calendar

Calendar is one thing everyone uses, and I used Cron Calendar (later acquired by Notion) a lot, and wanted a good replacement for Linux. Though I use calendar.google.com often on the web.

But the best replacement I found was Morgen (built in Switzerland) and is made for Linux first. It has a great preview inside the top bar too and timezones built-in.

![[img_Switched from macOS to Linux- 6 months in_1770740582450.webp]]

Time zones can also be activated by hovering on the time on the left:
![[img_Switched from macOS to Linux- 6 months in_1770740650778.webp]]

Daylight and Gamma light Adjustment

Sunlight adjustment like f:lux. Omarchy comes with one included right now, but I also used wlsunset with wlsunset -l 47.4095 -L 8.5514 -t 3500 -T 6500, that does the job well.

Hibernation and Suspending Computer

Hibernation and suspending is something that you take for granted on other operating systems. But on Linux it’s trickier, so it didn’t work out of the box. In the meantime, it comes built into Omarchy.

Presenting with External Projectors and Screens

Presentations and recognition of presenters and screens. I only had one presentation, but tried many external monitors, and Hyprland (which is responsible for recognizing screens hyprctl monitors) works just like macOS by auto-recognizing them.

Even better, I have shortcuts to make them automatically align at the right position. Or use hyprmon (one of the great [[TUIs]]) when I need to do it manually.

PDF Merger

PDF Arranger for merging multiple PDFs into one or rotating pages of a PDF. It’s open source and better than macOS Preview.

Need Anything More? Just Build (vibe code) it Yourself

If you need something, you just build it with [[Claude Code]] and integrate it into your laptop.

No need to ask Mr. Bill Gates or Tim Cook to integrate it. For example, I needed an edge light for video calls, or saw someone who had this. I liked it, not that I really needed it (but one day it might be helpful 😀). I’ve built a small custom tool that works out of the box with Hyprland for my future video calls.

Screen Sharing Works Differently

For example, screen sharing is not as straightforward because you get a very old frame to pick your windows or output or regions. On top of that, you usually need to pick twice and only the second pick will count. This was very confusing and I documented a fix and how it looks at [[Screen Sharing on Wayland (hyprland) with Chrome]].

But with the latest updates of Omarchy, that has also been solved and it works out of the box and looks beautiful now:

Others: Virtual Envs, Remote Desktop and Adding Printers

For virtual environments, I’m using Mise, as it comes pre-installed on Omarchy. Before, I used asdf.

Remote desktop to virtual desktops works great with xfreerdp3, which connects well to the Windows VM.

Need to import images from camera? Not as UI-driven as on Mac or Windows, but amazingly simple and fast with gphoto, see [[Import Files on Arch Linux (gphoto)]].

Adding printers might be needed at some point. This can be done UI-driven with system-config-printer - CUPS configuration tool. Or do it the terminal way with lpadmin, see [[Adding Printer on Linux]] for more information.

Running Microsoft Windows inside Linux

A big one is to run another operating system, in this case Windows, as part of your OS. E.g. I use Microsoft Office often, so I can quickly start up Windows with Office when needed.

The best part is that it uses only Docker, meaning easy setup, separated from my configs. It’s a single-click setup taking 15 seconds.

Installing and integrating seamlessly in a Docker VM works superbly with Omarchy. I submitted a PR to Omarchy to make this available to everyone. The built-in version in Omarchy (the first version) was done by me, and it was merged into core and is now available to everyone.

It’s now easier to run Windows on Linux than natively on a Windows machine 😉.

Finding the Right Hardware: The Reasons why not to Switch

Everyone knows the stereotypes about Linux. WiFi won’t work, Bluetooth won’t connect, constant interruptions. And beyond that, there’s the hardware fear, that you simply can’t match what Apple offers. A common sentiment:

I’m currently with this dilemma. I’m an experienced Linux user, but over the years gravitated towards Macs (especially M-series) and unfortunately they do make better hardware, at least for my use 🙈 I just can’t move to a machine with a much worse battery life, display, webcam, speakers etc. I know some good Linux-friendly laptops exist, but it’s still a downgrade, for me. If someone made better hardware, I’d probably jump over right away. Tweet

I thought the same. The great keyboard, camera, speakers, trackpad, battery. Apple just nails the whole package. But what I found is that I didn’t actually have to downgrade.

I started with a Lenovo ThinkBook 14 G7 ARP (AMD) with 32 GB RAM. Great build quality, beautiful look, and the keyboard surprised me, with much more travel and grip than the MacBook. See more on Part 1.

Once I realized this would become my daily driver, I searched for something more powerful for data engineering work and landed on a Tuxedo InfinityBook Pro 14 Gen10 AMD with 128 GB (!!) RAM, an AMD Ryzen AI 9 HX 370, and AMD Radeon 890M.

First impressions: super smooth, even snappier than the Lenovo. Obsidian and other apps feel a tiny bit faster. The 3K 500-nit display is stunning. Crisp, bright, better than my external 4K monitor. And it’s matte, which I’d forgotten I actually prefer. It works outside, no glare. The Lenovo’s anti-glare screen was equally great in that regard.

The keyboard has less travel than the Lenovo ThinkBook (which I really loved), more like a MacBook, which is fine but feels slightly cheap. I mostly use external keyboards anyway. The fingerprint reader is also missing, which I’d grown to love on both MacBooks and the Lenovo, where it worked flawlessly on Linux. The trackpad is smooth and great to work with daily, though palm detection caused some cursor jumping in the first days, not as good as Apple’s, but perfectly usable.

Battery life was a pleasant surprise. My Tuxedo (80Wh) delivered battery life comparable to my M1 Max MacBook. I spent a whole afternoon in the library and it was still above 70%.

With Omarchy, everything just worked out of the box. No WiFi or Bluetooth issues, speakers with sound, all good.

But it’s not perfect, by far. I get some strange popping sounds from the laptop, mostly after hibernating once or twice. Not sure why, and it probably shouldn’t happen. There are many other [[Notebook & Desktops for Linux]] to choose from, Framework being one, but choosing is still tricky, as chipset and GPU support on Linux matters, and you want something state-of-the-art.

Another side effect, as Kevin says of not having an expensive Macbook:

My favorite part aside from customization is just that i don’t care about my machine at all: it gets lost? breaks? stolen? i get a new machine, run 1 command and everything is back exactly as i left it. Macs on the other hand are expensive to buy and repair which makes people worry and worry = less peace of mind.

[!example] Follow the evolution on Social Media

The whole story I documented in threads on Twitter and on Bluesky, follow these to see the history and events in they happened.

Conclusion of Using Linux for 8+ Months

After using Windows since 2003 and macOS for more than 15 years, how do I feel after 8 months on Linux?

Things mostly work great, but need a little tinkering to begin with, or work differently. The biggest difference, which I like a lot, is a more terminal-native workflow. Closer to the command line. Using lots of [[TUIs]].

When I started, I just wanted the same as I had on macOS. After getting familiar with the new environment, with all the small utilities, tools and programs Linux has, I got many more tools to choose from. Sometimes much better, though terminal-based, but fast and direct. Sometimes you obviously miss a tool that has no replacement (for me still Snagit).

Besides the obvious (terminal-native, best-in-class Tiling Window Manager with Hyprland, no-latency navigation), there’s something harder to put into words. The OS is what we use every day, so when you can quickly fix or change a small thing to give you more joy or more productivity, it might just put a smile on your face whenever you use that feature. At least it still does for me. And since all my configs live in dotfiles and my data syncs externally via Filen and Obsidian, setting up a new machine is a single command.

What I Thought I’d Miss vs. What I Actually Miss

Before I switched, I thought I’d miss all my Setapp, my MacBook hardware, and the stability of just working. Most of my apps work on my new machine and even better software-wise, so I’m still quite happy to have made the switch, even more so watching macOS get slower with each install without real benefit (looking at the Liquid Glass update) and Windows stuffing Copilot into Notepad and recording your screen with Recall.

What I actually miss are simpler things: the stability of having calls everywhere all the time with Apple reliability and inbuilt mic/speaker/camera. A crash because the GPU is fails¹ after hibernation right before an important meeting. Hibernation and suspending was quite a battle to get working, but it seems to just work now.

What I like about Linux: it might not work out of the box for every laptop or every program, but you can actually fix it, and from that moment you know the problem, you learned something about computers, and the error will not appear again. Unlike other operating systems that change stuff you set in settings for a reason, only to learn that certain updates turned that checkmark back on.

Tinkering and Troubleshooting: Not Everything Just Works

Probably, without [[Claude Code]], I wouldn’t have made the switch, or I would have made it, but probably wouldn’t have stayed. When something happens, e.g., a crash out of nowhere, I just open Claude and say: I had a crash, I am running Arch Linux, please check the logs what went wrong. And what I get is a full analysis of what went wrong, some fixes and suggestions. Knowing that Linux has 100 different setups, different drivers for every hardware, this is a non-negotiable lifesaver.

With Linux you also have to troubleshoot bugs, but at least it’s free software and open source, and honestly, they seem even less frequent than with commercial, paid products these days.

If you’ve read this far, thank you. What’s your experience, are you thinking about switching, or already on Linux? Let me know anywhere on Bluesky, or Twitter.

Again, if you want to watch a full video workflow, check my short video about it: Omarchy Arch Tiling Window Workflow (macOS comparison) - YouTube. Or my macOS workflow as a comparison and what I switched from.

Appendix: Troubleshooting and Things That Didn’t Work So Well, or I Had Already Fixed

GPU Crashes (AMD Radeon 890M). This was the biggest recurring issue. The GPU’s MES (Micro Engine Scheduler) would become unresponsive and crash the entire system, triggered by Brave browser, Google Meet video calls, and Kdenlive video encoding. The root cause is that the Radeon 890M (gfx1150/RDNA 3.5) is still very new, and driver support on bleeding-edge kernels (6.17–6.18) is immature. Solutions included disabling hardware acceleration in Brave (brave://settings/system), adding kernel parameters (amdgpu.gpu_recovery=1 amdgpu.noretry=0 amdgpu.ip_block_mask=0xfffff7ff), and considering the LTS kernel as fallback. The community is tracking this on Framework forums and AMD’s GitLab.

Keyboard Freezing After Suspend (Tuxedo). The internal keyboard would stop working after suspend/resume cycles due to a firmware bug in the keyboard controller (i8042). Fixed by adding i8042.nomux=1 i8042.reset=1 i8042.noloop=1 i8042.nopnp=1 to the kernel command line in /etc/default/limine and regenerating the UKI with sudo mkinitcpio -P. Shared the solution on Reddit.

Hibernation Not Resuming. After suspend-then-hibernate (triggered by closing the lid), the laptop wouldn’t resume and required a fresh boot. The cause was missing resume= and resume_offset= kernel parameters. Omarchy’s hibernation setup script added the mkinitcpio hook but never added the actual kernel parameters. Fixed by calculating the swap offset (sudo btrfs inspect-internal map-swapfile -r /swap/swapfile) and adding resume=/dev/mapper/root resume_offset=<offset> to /etc/default/limine. Documented the fix in this Omarchy issue.

Thermal Throttling (Lenovo). The Lenovo ThinkBook would hit 99°C and become unusable during video calls. Turned out the bottom intake vents were blocked when the laptop sat flat on a desk. Simply elevating the laptop dropped temps to 73–77°C and performance was completely fine, even running stress tests while screen sharing. A laptop stand solved it permanently.

WiFi Speed Drops (Tuxedo, Intel AX210). Speeds dropped to 2–72 Mbps after a system update. Root cause was a bug in linux-firmware-intel version 20251125 that caused the Intel AX210 card to negotiate very low RX bitrates. Fixed by downgrading to the October firmware (sudo pacman -U /var/cache/pacman/pkg/linux-firmware-intel-20251021-1-any.pkg.tar.zst), disabling WiFi power save permanently via a systemd service, and adding IgnorePkg = linux-firmware-intel to /etc/pacman.conf until a proper fix ships.

Keyring/Brave Re-login on Every Boot. Brave asked for login credentials after every reboot because the gnome-keyring file kept getting corrupted. This was caused by SDDM autologin. Without entering a password at login, PAM can’t unlock the keyring. The ultimate fix was launching Brave with --password-store=basic in the autostart config. Documented in this Omarchy discussion.

Sudoers Misconfiguration. While adding a NOPASSWD rule for a keyboard-switching script, I accidentally broke sudo access entirely. Had to boot into recovery/single-user mode to fix /etc/sudoers. Lesson learned: always have sspaeti ALL=(ALL) ALL as a separate line and be very careful with visudo. Documented the recovery process in an emergency recovery guide.

Screen Recording VFR Issues. Omarchy’s screen recorder (gpu-screen-recorder) produces variable frame rate videos by default, which Kdenlive can’t edit properly. The fix is adding -fm cfr to the recording command. Additionally, Kdenlive’s VAAPI hardware transcoding crashed the GPU (same MES issue), so software encoding (libx264) is needed for now.

CPU Fan Noise When Plugged In. The system switched to “performance” CPU governor when plugged in, causing constant full-speed fans even at low load. Fixed via powerprofilesctl set balanced or through Omarchy’s built-in power settings menu.