The Act of Creation

The act of creation is an outlet. It gives joy to us when we create something out of nothing, we block out anxiety or boredom.

Creating should be done like:

• a kid in mind: effortless, exploring your thoughts, and seeing where it leaves you, as Picasso said: He wouldn’t have bothered to start a painting if he knew the outcome.
• a play¹: flawless and free.
• calm like the water: flowing wherever your mind is going.
• to fulfill yourself: by portraying your never-resting thoughts. Share it with the world. Create with joy and love in mind.
• follow your train of thought: see where it leads you. You start somewhere, on an unfinished emotion, a task, or anything to ponder on.

Creation, and especially writing, can help resolve the unresolved.

Creation to Make a Difference: Joy and Laughter

Creating is also about making a difference. If nothing changes in you, the people, not enough value was added to your creation. Any creation should trigger something. But foremost, the process of creation should change something in you, some release, a good feeling, it must feel right. If it does nothing, an empty feeling, the creation might not be there yet.

Create with love and empathy in mind. Liking someone is one thing, but giving love, caring for someone, and integrating that into your work is the ultimate way to create.

Create joy and laughter. Speak the language of people, make them happy by creating enjoyment.

Following Our Emotions without Boundaries

When we follow our emotions without any boundaries, following our instincts, we get into a state of deep focus and deep concentration where all thoughts can be resolved.

Creation is our inner [[Gut Feeling|Instinct]]. If we follow and nourish it, great things can arise from it. Treat it as an outlet for your body and mind. Go with the flow, follow along. Create for the sake of creation.

Learn for Life, Follow Your Own Uniqueness

The act of creation is following your own life rhythm, for example, the [[Pathless Path]]. It is the ultimate form of connecting with yourself.

All creation is unique, unique specifically to you as the creator, but also (hopefully) to the reader, viewer, or consumer of your work. Your creations live long after your death; others will connect and create new works from them. It’s giving joy after you’re gone.

Learn for Life, as I like to say, has been my earliest mantra when starting any of my creations shared online. The common definition of success is reaching a desired outcome. The Scientist’s definition of success is: if you learn something new, you haven’t failed, as Anne-Laure Le Cunff says on How to Design Tiny Experiments Like a Scientist.

For example, a scientist defines it like this: if you say you want to write more, instead of just writing, you define ‘I will write for at least one month, or at least 10 articles. And then you don’t stop before that. Success is the experience, not the outcome.

The Outcome of Creation

The act of creation is a slow process that needs time and experience. A slow life, an [[The Ordinary (Boring) Life|ordinary, boring life]] even, to focus on the details, using the craftsmanship refined over the years, and retrieving joy from the process of creation.

What matters is the quality and outcome we are proud to release to the world.

Gifts Shared with the World

You create for yourself with no return in mind, you just share it as a gift for the world, for anyone to consume.

Creation thought of as a gift is an easy way for you to create without the burden of pleasing people, as it’s take-it-or-leave-it, like a gift. No strings attached, just making gifts.

A Curious Mind is Mostly a Subconscious Mind

When we create, most of our subconscious is driving the thought. Just let it cruise and see what the outcome is. Use the conscious mind for fixing errors and making sure the sentences make sense later, but don’t start with it.

The Spark of Joy Philosophy

Different flairs for designs to add ease or user delight to your creation. Similar to the “Spark Joy Philosophy”, which is very fitting when creating for the enjoyment of the reader.

To me, creation sparks joy for myself, it’s the outlet for me to release and organize my thoughts. It avoids the [[shallow happiness]] that I get from social media, binge-watching Netflix, or other brainless activities on the phone or TV. Sure, there’s time for that too, to calm down, but if we are not careful, the algorithms will take over and we default to always choosing the easy choice.

The Reward is Long-term

Creating is hard, there’s friction, it does not directly work, but the reward is long-term, with [[deep happiness]] and a deep flow state: “It’s magical that they just tried to be there”, which is the highest form of happiness artists get (see The Artist’s Way) as we discovered in Finding Flow, with escaping digital distractions through deep work and slow living.

I hope you find your outlet for creation, an act you can partake in to develop happiness and joy for yourself, getting into that deep flow and having a slow process where many small gifts of creation can be shared with the world.

This article was created with the inspiration of reading How to Live by Derek Sivers in Cavallino, Italy.

For text editing, we have Vim motions that help us navigate a text document with 1000s of shortcuts, but because there is a grammar, we do not need to remember them all, but learn the structure of the grammar and combine them. But what if you work in data? What if we could have the same for data, a grammar for data engineering, or a language that defines it?

Expressing our needs declaratively and decisively? Also, expressing it in a way that leads to reproducible outcomes, or works with multiple parts and execution engines already out there. This is what we will discuss in this article. How existing tooling, such as Ibis, provides some capabilities, and how xorq extends them by adding full lineage and transparency for humans, with included executable memory for useful tabular data, all manifested in a single git repository.

Expressions for Data Engineering Workloads

Having a grammar for data engineering means we can express the workloads in a declarative manner, and then be sure we can deterministically reproduce and apply that exact definition.

It’s similar to the concept of a Declarative Data Stack I introduced a while back, but it gives the stack not only configurations but also a language with in-built manifestation and execution engines.

In the above image, we see:

1. How to express (write) our transformations and business logic. It’s the context of every ML or DE pipeline.
2. We can build the expression into a manifest that has a unique hash, runs input validations, tracks lineage, creates a deterministic cache, and produces a human-readable expr.yaml you can diff and review in a PR.
3. Lastly, we can execute it in any execution engine with the same manifest.

This is hugely powerful and separates the concerns of defining logic, verification in the manifest step, and execution as a composable data stack, as Wes McKinney called it, with multi-compute engine possibilities.

How the DE Language Works: Different Expression Types

Every grammar starts with nouns, and here the noun is the source, a node that holds data but carries no transformation yet. It might be an in-memory table, a registered connection to a warehouse, or just a lazy pointer to a file on disk that hasn’t been read. They’re simply referenced, the way a noun refers to a thing before any verb acts on it.

The verbs in our language are transforms such as filter, select, mutate, aggregate, join, order, limit. Each one takes a source (or another transformed expression) and returns a new, immutable expression. You do not mutate anything before it, only describe what should happen next.

Looking at a definition such as .filter(...).aggregate(...).mutate(...), we can see this as a sentence. The moment a verb is applied, the expression stops being a plain noun and becomes a statement, a description of “data plus what should happen to it.” But the sentence isn’t spoken yet, it stays inert, fully composed but unexecuted, until something finally asks it to run. That’s the deferred part of the grammar: writing the sentence and saying it out loud are two different acts.

There’s a third part of speech worth naming: the template. Instead of writing a sentence about a specific noun, you can write one about a noun’s shape, a schema with no rows behind it. A template says “given something with a column of this type, here is what I’ll do to it,” and only later gets bound to an actual source, at which point the placeholder resolves and it becomes an ordinary statement again.

And we have modifiers that ride alongside a statement without changing what it computes. They’re small tags of metadata that say “this expression also represents a fitted model” or “this is a saved reference to something else.” It’s like a footnote with additional metadata that doesn’t change the surface meaning, but adds context for later use.

This analogy makes the grammar compose the same way regardless of which engine eventually executes it. There are more parts, but with just these four, noun, verb, template, modifier, you can read (and write) arbitrarily complex data pipelines the same way learning a handful of verb-and-object combinations in a text editor lets you compose arbitrarily complex edits.

[!tip] Avoids building “Inner-Platform Effect” with repeated tools
With this grammar, we can avoid repeatedly implementing the same logic we already have, but manifest and express our logic once, and reuse it with different execution engines, exactly what Ibis and xorq allow. Similar to what the inner-platform effect means for software best practices.

Why a Grammar is Really Good for LLMs

Having a grammar is really good for LLMs, too. It helps them first to declare data artifacts and second to execute them reproducibly.

On top, expressions can be LLM-agnostic, and we can interchange the LLMs we use just with an expression. Also, the chart is just an expression, or the data catalog and the metrics.

Model Once, Represent Everywhere: Expressing the Full Data Stack with a Single Expression

Like UDA (Unified Data Architecture) from Netflix, we define our expressions once and represent them everywhere. Netflix built UDA to solve duplicated models, inconsistent terminology, and siloed systems, where the same concept like ‘actor’ or ‘movie’ gets modeled differently across teams, with no shared foundation. Their answer was a full knowledge graph with a metamodel, making the conceptual model part of the actual control plane.

Not everyone needs Netflix-scale tooling, though. For a code-first approach, xorq gives you the same core principle: define once, execute anywhere by writing a declarative Ibis expression, serializing them as content-addressed YAML artifacts, and running against any supported engine, fully reproducible.

The difference worth noting: UDA is a semantic layer defining what data means across systems. Xorq is a computational layer defining what transformations do across engines. Both reject the same anti-pattern of re-implementing the same logic for every system.

Entering Xorq: The Horizontal Data Architecture

Xorq is an executable memory system for tabular data that works horizontally across your data stack, supporting everything from discovery with a catalog to defining transformation logic to modeling.

It has declarative transformation (Pandas style), and you can build ML pipelines and prepare data with its semantics in a single stack that is not vertically integrated, but horizontally integrated, giving your agents a catalog of executable pipelines and turning short-lived agent work such as wrangling scripts, sklearn pipelines, ad-hoc tables into durable, composable, executable artifacts that any future agent or human can discover, reproduce, and reuse.

The horizontal data stack shows what Xorq brings to the table. Xorq’s origins started from a git-native semantic layer, for data analysts out of college, to build semantic models for a living, to make their lives easier.

From point-and-click tools, dragging tables and drawing joins manually, only to add more reporting tools on top to create pixel-perfect reports. Also performance-wise, it didn’t scale, meaning we needed cubes to make it faster, adding another layer of complexity.

And there was no lineage that shows from source to dashboard. The question asked was: “what if we could do this end-to-end data engineering workflow locally?”. This is what the horizontal data stack and xorq are providing.

To add semantic layer capabilities, Julien Hurault and Hussain built the Boring Semantic Layer + the Xorq catalog, providing a semantic model you define in Python, check into git, and query from the CLI.

Compressing Logic into a Single Executable

Compression of a full data stack into a single executable is hard, but xorq tries exactly this with the help of Ibis, git, uv, and DataFusion.

The design choices of xorq showcase even better what it is, and what they enable:

• Ibis as expression layer (v9.5.0+, partial): Declarative dataframe expressions compiled to multiple backends (xorq supports a subset of the Ibis API, not the full surface)
• Git for state and storage: The catalog is a git repo of entries with git-annex support for large files
• uv for reproducible environments: Each entry ships with a wheel and pinned requirements.txt.
• DataFusion for embedded compute: Pipelines execute in-process with SQL and UDFs

Composable Data Engines

Another big advantage of expressions and having a grammar for data engineering is easily switching between backends, with no change to the transformation or business logic. It’s just defining the backend from Apache Arrow Flight to DuckDB or any other engine.

We write the definitions and express our tabular data and computations. The engine, in this case xorq, can build it into a manifest file that is deterministic and hashed.

Xorq uses Ibis as the expression layer for single-backend logic, then builds the cross-engine expression tree into a serialized YAML artifact. When moving data between backends, xorq transfers Apache Arrow RecordBatch streams between them—each backend acts as a RecordBatch transducer. No CSV serialization, no JSON encoding needed. This makes backend switching fast and memory-efficient. Write declarative Ibis expressions that run like a tool—xorq extends Ibis with caching, multi-engine execution, and UDFs.

Here’s an example of using DuckDB and Postgres in conjunction:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

import xorq.api as xo

# Connect to engines
pg = xo.postgres.connect_env()
db = xo.duckdb.connect()

# Load data from different sources
batting = pg.table("batting")
awards = xo.examples.awards_players.fetch(backend=db)

# Filter in respective engines
recent = batting.filter(batting.yearID == 2015)
nl_awards = awards.filter(awards.lgID == "NL")

# Move data to postgres for join
result = recent.join(
    nl_awards.into_backend(pg),
    ["playerID"]
)

result.execute()

Move data between different engines within a single expression using into_backend(), here Postgres and DuckDB

You can see how easily you choose your most optimized execution engine, whether in the above example choosing DuckDB for filtering recent batting and using Postgres to filter NL (National League) awards, and joining the two with the Postgres engine.

Engines supported by xorq as of now, with the ability to move data between them, are (check Supported backends for the latest):

• Embedded: DataFusion, DuckDB, SQLite, Pandas
• Warehouses: Snowflake, Databricks, Trino, Postgres
• Lakehouse: PyIceberg
• Arrow Flight: GizmoSQL (DuckDB over Arrow Flight SQL)

Cross-Engine Expression Tree

With different engines supported, we can use the compressed single executable logic across engines. We can build expression graphs before executing them, which works like this, with one expression, many engines:

1
2

expr = penguins.into_backend(xo.sqlite.connect())
expr.ls.backends

The output of building a cross-engine expression is a directory containing your serialized pipeline with a unique hash identifying each build and its artifacts and expressions. When executed, the output is the resulting object or data.

And the expressions are tools, Arrow is the pipe. E.g., a Unix pipe streams text between small programs. Xorq pipes Arrow streams between expressions: unix : programs :: xorq : arrow-transforms

That executes like this:

1
2

In [6]: expr.to_pyarrow_batches()
Out[6]: <pyarrow.lib.RecordBatchReader at 0x15dc3f570>

This is quite short and potentially abstract to understand when never used, but we will go into more examples and details in another article.

A Shared Language for Data

This article introduces a new way of describing data transformations for machine learning or data engineering pipelines in a direct and simple way that works locally with any execution engine, without changing the code itself.

It’s a good place if you need a trusted harness for a data engineering persona. We can define once and use it with the engine that works best for your workload and data engineering environment.

We had a look at how we write -> manifest -> execute with xorq, its advantages, and why you might use it for modeling once and representing everywhere. By adding AI agents to the mix, which help us pull the right lever, instead of bigger, more expensive models or more tokens, we improve accuracy with more semantic understanding, with a grammar the model can learn and apply, even pre-manifest before execution, and run them deterministically every time. This is a huge addition to working just with agentic Skills files that are free-form Markdown and pull data all over, or are not defined precisely enough. It’s all about having high-quality context in the right format, with a clear definition where humans and AI agents can interchange and help each other.

There’s a lot more to come, with showcasing the horizontal data stack and the use cases it supports, how we build expressions versus running computations, and how data catalogs are integrated into the picture, too.

–

Check out xorq code and star it on GitHub, or read more behind the scenes at Xorq documentation.

They also have a macOS desktop app coming up that does it all in one unified app, geared towards non-technical users. Join the waitlist for that.

Full article published at xorq.dev - written as part of my services

The danger isn’t only a crash or an error message, but a wrong number that didn’t break. It might be a clean query, but it introduces duplicated rows.

In this article, we go through the three levels of AI agents in data engineering, how to structure projects so the AI delivers its best outcomes, and how dedicated agents with a deterministic core help us build higher-quality pipelines — ones we can actually trust. And we look at a practical example of how it works with a blast radius analysis.

The Three Levels of AI Agents in Data Engineering

Why should we use agents for data engineering? And at what levels can agents help us productively? As LLMs will always have some error tolerance, as humans do too, we need a way to be more confident in producing the code.

Chat-phase, Autonomous and Dedicated Tooling

There are different levels of confidence and levels on which the agents can help us.

1. The initial chat-phase: the development where we prompt Claude or ChatGPT. The model tries to understand the context based on what it has access to. It takes a decent amount of tokens, as it needs to scan everything from scratch.
2. The autonomous approach, where Claude Code or Codex also have access to the tools humans have, mostly the CLI on the terminal, making it possible to query Postgres with psql or read from S3 or Parquet with DuckDB to verify queries and data. A much higher quality outcome.
3. Dedicated agents for the task at hand. E.g., for data, the tools know dbt or know how to transpile SQL code deterministically, meaning not from training data only, but with an actual tool that does it much faster and more reliably. Built-in checks and features a “general” agent can’t provide.

Ideally, we’d want to always use the dedicated tools, but there isn’t always one.

Where in the DE Lifecycle Each Level Actually Helps

BI Dashboards vs. Plumbing the Data Pipelines, or Creating Source Ingestions, or Maintaining? For data engineering, the question is not only if there is dedicated agent tooling, but also on what part of the data engineering lifecycle AI agents can help data engineers and analysts the most, and potentially even domain experts?

The lifecycle contains the ingestion part, ETL, or understanding the business in great detail, or is it just to visualize the result? Or should it cover maintenance in case of overnight ETL errors, or the full data lifecycle?

In general, before we go into more details later, agents can help us on the full cycle, but it always depends on who you are and what role you play. Building from scratch with no knowledge or seniority is dangerous. Why? Because they can’t verify if the produced code is correct. Okay for a side project or a proof of concept, but not for actual production.

What’s the Engineering Discipline for Working with AI?

There’s also a part that is less technical, a way of guiding the agents in the right direction. Especially if we want to safely use it in large projects or organizations, we can’t just let it run without guidance.

For that we need:

1. clear project structure in which the agents can flourish. The more is given, the fewer tokens are used for this work, and it will be more aligned across the project. (Another reason a deterministic workflow such as uv init is best, because it will always be the same).
2. build with clear instructions (agentic skills, superpowers, etc.) on how the tools are used (basically providing CLIs and API documentation). This is the bulk of the work anyway. That’s the data architecture, the brainstorming with fellow humans before you build something, instead of missing a key insight in the beginning and then letting the agent run down the wrong path. Also, be realistic: prompting “be correct” or “use state-of-the-art” won’t make it more correct or more state-of-the-art than the model was trained on. So if it’s a rather new architecture, it’s a must that you provide these links and hints.
3. set up the project in a modular fashion, so the agents cannot break the whole project if they make a small change, so you don’t end up in a scenario like dependency hell with everything dependent on each other.
4. use a declarative approach, with descriptive configuration that says the what and not the how, so that you can collaborate on these configs with the agents, version them, and easily revert or change something, as well as decouple the implementation logic from the actual business logic.

With these steps, you can get the best out of the agents of today. I’d say the model matters less, but the structure does, and as Mario says, so does the workflow approach. For example, extensively plan (the process before writing a single line) and correct the model before any implementation that could lead down the wrong path is written.

Also, don’t overthink it. But this is only the workflow and learning the soft skills and discipline of working with agents. How does that look in a real-world project?

[!note] The key is to get use out of AI, not to get more work.
E.g., most developers used to think about the problem. Today, most drown in PRs. When the AI tooling gets better, AI can provide more quality code that is correct, that needs less review or fewer iterations, which means fewer PRs and less work for the developers to go through.

The Correctness Layer for Data Engineers

A key insight is that AI agents should support the “human in the loop” for correctness, or a correctness layer. And rather than making more work to verify more code, we should be confident in the process and know that the code it produces is verified and ultimately correct.

But how do we get more “correct” work and a layer in which we can verify it? The biggest argument is a deterministic-validation architecture in full. E.g., Altimate Code splits the agent into a probabilistic layer on top and a deterministic Rust/TS layer underneath that does the actual SQL ops such as parsing, validating, and equivalence checks, so that the agent itself never has to be trusted on those questions.

Altimate Code, for example, is built on a probabilistic agent, deterministic harness, and deterministic core. The probabilistic agent with the LLM does the creative work of reading intent, picking a strategy, drafting SQL, summarizing results, and recovering when something goes wrong.

Below the boundary sits the deterministic harness, a TypeScript layer that intercepts every tool call: a dispatcher checks hasNativeHandler before the call runs, and routes it either to a native, deterministic handler or back to the model. Those handlers don’t reimplement logic themselves, they call into the deterministic core, a Rust engine (altimate-core) that exposes SQL operations as pure functions over ASTs and schemas, wired in via napi-rs bindings. Parsing, validating, transpiling, checking query equivalence, diffing schemas, extracting column lineage, diffing rows across warehouses — all of it runs sub-millisecond, and all of it returns the same answer on the same input, every time.

Like a compiler, the agent never decides whether two queries are equivalent or a column exists upstream. Instead, it calls a function that proves it against the parsed AST and the schema, the same way a type-checker proves a program compiles rather than guessing.

That’s the distinction that makes the output easier to review, as factual checks have been run and the output is either correct, or there’s a bug that it can fix directly. The rest a human can re-verify. On the dilemma of having stopped to hand-write code and approving it faster than humanly possible to check, you can also read more at You Are the Trust Layer.

[!note] There’s another factor, being wrong
Bare agent use might be cheap, but only until they’re wrong, and then the cost is unbounded.

Improvements for Better Usage of Tokens

Altimate, or data engineering agents that have deterministic functions and integrated understanding of how to work, can help you save tokens and be token lean (the opposite of [[tokenmaxxing]], which is popular on Twitter/X, using as many tokens as possible and having an agent running at all times). Because in large enterprises, token costs are a real budget point.

To slow down the tokens, an easy trick is to instruct the model to use fewer tokens and words itself - caveman is a good example of that, but you can also add a singular prompt to your CLAUDE.md, Codex, or model of choice in combination with Altimate Code.

There’s a second, less obvious cost: the token itself isn’t a stable unit. When Anthropic shipped Opus 4.7, the same prompt that cost X tokens on 4.6 started costing roughly 1.4X (same input, same answer, more tokens, same price per token).

Altimate on The Great Token Heist of ‘26 makes the case that “cost-per-token is the wrong number to optimize”, since the meter itself can move with a vendor’s next model update, and what we should track instead is cost-per-task. I fully agree, and this is where deterministic function calls work around that volatility by not using a model/tokens for every task, making it less expensive.

Typical Use Cases

In this chapter we go through typical AI agent use cases for data engineering.

There are many of them. You can use them to educate yourself or your team, build production data pipelines, build data apps, and visualize your data in new innovative ways (usually HTML web pages with React and other JavaScript frameworks). But in general, the use cases fit into these approaches:

1. Start a new project from scratch example: Building a data landscape with more open source.
2. Extending an existing project or data warehouse: Adding new data pipelines.
3. Maintaining current setup: Update and verify it still works when changes come in.
4. Migration: Migrate from one database or tooling to the next.
5. Finding the Blind Spots: Two similar-sounding IDs might be wrongly used for a join, or missing data in a column that got missed in a nightly load, or anything in between. If agents can do these checks, that would be super beneficial. With more access to CLI, Model Context Layer, and deterministic tooling, these things are truly possible.

Below we go through extending and changing an existing warehouse with a change of column, and using Altimate Code to give us a Blast-radius assessment.

Showcases: Blast-Radius Example

A Blast-radius refers to the potential extent of damage. For example, before you knock down a wall in your house, you want to know if there’s plumbing behind it, electrical wiring within it, or if it’s holding up the floor above.

The same is true for a data warehouse or a data project with lots of ETL. For example, if a data engineer cleans up the table fct_orders by joining orders to order_items and summing order_total. It compiles, the dbt tests pass, nothing errors. But the join changes the grain, so any order with several line items now gets counted once per item, and revenue quietly inflates.

It’s best to know, before you rename a column or add a new join, the downstream (data that comes after the current task) dependencies to the dashboard — that’s what the blast-radius report does.

With Altimate Code we can achieve this. Before any change goes through, it maps out the full impact automatically and produces a detailed blast-radius report with what will break, what’s safe, what needs someone to sign off, and also performs the changes. Here is what this looks like:

Rename and Change Columns and Logic

As an example, in this prepared ecommerce repo with different DWH layers such as staging -> intermediate -> marts, I prompted this request to change unit from cent to dollars:

It recognized the dbt name and invoked dbt-analyze automatically:

It gave me a full Blast-radius report and the impact my changes would have on the project:

Including semantics only, to point out what’s safe and what’s not:

With a fixed order to address breaking changes, semantics and docs, and intentionally untouched:

Notice, I hadn’t said anything about blast analysis or using dbt-analyze — it did it on its own, ran dbt, and analyzed it deterministically.

This shows how Altimate Code looks behind the walls of data engineering, just like blast radius analysis.

If you want to see another example and a full blog post on Blast Radius, check out Blast Radius Analysis Using Altimate Code, and what Altimate Code did as in the video. Or Altimate provides many more examples and Showcase on their website such as Migrate SQL Server to Snowflake with dbt or showing how to resolve An Upstream Schema Changed.

[!example] Connect a model to Altimate
Make sure to connect to a model with /connect and choose an existing subscription with API credits, or any other subscription. I used opencode zen for my example, which includes e.g. Opus 4.8.

Correctness Over Confidence

I hope you got a better understanding of why AI agents can be genuinely useful, especially when provided with the right tools and applied with the right discipline.

You’ve also seen how deterministic tooling, purpose-built for data engineering and analytics problems, gets you both better correctness and better token economics than general-purpose agents alone.

Coming back to where we started: not every task needs a level-three agent. A quick chat-phase agent is fine for exploring a dataset or drafting a query you’ll review yourself. But the moment that output touches production or serious work, a dashboard, a nightly job, a number someone makes a decision on, you want the deterministic core underneath it, not just a model that sounds confident.

That’s the gap Altimate Code is built to close. It runs on deterministic functions purpose-built for DE workloads, it’s open-source via the OpenCode TUI, and for teams wanting more, there’s Altimate Studio — a paid, multi-agent platform with extras like warehouse cost optimization, dbt development acceleration, and migration tooling.

–

Check out Altimate Code, it’s free and open-source. Give them a star if you like them, and find more information on their docs and new website.

Full article published at Altimate.ai - written as part of my services

So what is important when taking notes? And how you might take notes too? First, the book [[Smart Note Taking]] by Sönke Ahrens helped me tremendously to understand the basics, that was after I was taking notes already for 10 years, but the principles of it being fine to delete notes, to go from literature notes to evergreen (though I changed that a bit for my liking), were two things that helped me a lot. To see the notes as a system of thoughts, more than just notes. From dumping all my thoughts into a blank file in my text editor of choice to sorting them, and getting clarity from it. And that writing is not the outcome of thinking, it is the medium in which thought occurs. This has grown since then, even more.

But if we think about it, we can also make it a hugely scientific and philosophical, but on the other hand, it’s just ideas we have in our heads to:

1. Write down in a way we don’t forget, and more importantly
2. find them again in a year or two.
3. Also, do not write only a couple of words, as later you might not know exactly what you meant, but also don’t write too many details, or worse, copy a full article.

The key is to pause for a second, and think about it, and then write it down in a sentence or two. I usually use listings, so I know it’s kind of a brainstorming, if I want to go deeper, I can just indefinitely indent and go up again. So I can, if I want, write a full outline, add new thoughts from the idea I just heard, link it to other ideas and notes in my vault.

But number two is harder than it looks to write it in a way to find it again easily. That took me a long time to optimize, and only changing of system from OneNote to Obsidian, and more so from file organization to connected notes, or also called [[Zettelkasten]]. This means, during the writing, I didn’t need so much thought where to put the note, is it in business, or personal, should I create a new folder, or do I need to search all my notes if there’s already a similar folder, to just open a new note, doesn’t matter where it is, directly type, link it to at least one or two other notes (so I find it later if I forget the file name based on backlinks) and give it a name that I might search for in a year. That is crucial, as I almost exclusively search all my notes via file name, the name is key.

Luckily [[Obsidian]] helps us with duplicated names, because if you open double brackets anywhere with [ [ and start typing, you find all notes with that name, so you don’t accidentally create a note twice. And for the finding part, if I know I have written once about it, but I can’t find the name, I search for the content with search or other mechanisms in Obsidian. But once I’ve found it, I directly rename the note to something I was searching initially, as this means to me, the name was not good enough yet, or add a keyword. So next time I can find it immediately.

The Process of Note Taking

There are different stages of note taking, the first part of capturing your note that we just described. But then there’s the refinement and the publishing phase. Tiago Forte, for example, calls it CODE (Capture, Organize, Distill, Express).

While you can add a lot of work for yourself with telling yourself you need to refine your notes, take time every week to process the latest, I do none of that (scheduled or planned). I believe the note taking should help me and not add more work to me. Sure, I refine them, and a lot actually, but it’s based on usage.

I look for signs. For example, I come back to a note over and over again, or I looked up a note two three times recently to understand a concept better, or I linked to a note many times. All of these are signs that this note needs more attention, potentially even splitting it, really refine, and then potentially publish on my public second brain (I can just add #publish, and it lands on my website), or write a full blown blog post on my blog website.

The Key of Format

One thing that unlocked so much more of the sharing part, and joy of using my notes, was the switch from the lock-in format in OneNote to an open [[Plain Text Files|Plaintext Files]] file [[Markdown]]. These files are just text files locally on my machine, meaning I can open them with my [[neovim|Text Editor]] or [[IDE]], or do a search and replace on all my files. Back them up easily, and also notice they almost use no space, as just text files. And the best part, I completely own them, so all my ideas and work I put into them, I will own forever as long as there will be computers. They will not die with the tool such as OneNote, Apple Notes or Notion, once they are gone, your notes are gone too, or if they deactivate your account, byebye all your notes.

Blogging

But the not foreseen effect, was that when I migrated my blog from WordPress ([[Rich Text]] interface) to a [[Static Site Generators (SSG)]] with [[GoHugo]], that is just Markdown. Now instead of needing to reformat, to upload somewhere in an online form, and constantly need to sync them between each other, say I updated my notes or fixed some grammar. No, now it’s essentially the same note. Obviously, the blog post needs a title, a hero image, maybe a description and some more attributes, the plain note doesn’t need.

But once I noticed this, I upgraded the workflow quite a bit. And I added my [[Quartz]] based second brain to my website (see ssp.sh/brain), and there I had the goal to do zero edits, and just hit a button and publish. And to be honest, that’s what I do for more than 6 years or so.

I add #publish to my note, and run a script (it ended up not being a button :), and it’s on my website under new recent note. Including all the backlinks I added via Obsidian and formatting, links title and everything. I even added an [[Dynamic Feature Images|automatic hero image generator based on each note]], and updated a bunch of more advanced things with the frontmatter that Obsidian later added too, same as my GoHugo had already, meaning the format aligned even more, and is even more the same!

What I’ve Learned

So what I’ve learned? It’s really important to remove friction, to make it so that you can use and enjoy it. Maybe for some it’s needed the friction of opening a web form and typing in there, same as some like writing the email newsletter in an email client, to feel the pressure of the readers more, than just in a text editor - which I also implemented in [[neomd|my email client]] and newsletter in newsletter-rss, I can either run a script and start an email and upload it to my email newsletter, or write it to newsletter-book@ssp.sh and my email client will create a new campaign in my listmonk and send it out 30min later.

But that is what I was saying, the pressure of writing in the email or WordPress editor directly can be a help, but if you want to [[Learn in Public]] and share what you learned more often, I think removing friction is key. Also to overshare some times, when the note is still very rough. Because people might give you feedback, that helps your thinking or gives you an angle that you never had before.

TL;DR

1. Take smart notes
2. Use connected links instead of folders to break out of the 1 dimensional world of folders, especially if you write or want to learn. (it takes time though to adapt)
3. Choose an open format, I highly recommend Markdown. Also read [[File Over App]] as Kepano said, the app might not be here in 5 years, but your notes will.
4. Publish notes to learn, make the process as smooth as possible, no formatting or copying/duplicating of your notes needed.
5. Have fun, and enjoy the process. Especially with AI agents, choose a framework like GoHugo or Astro that builds on Markdown and plaintext files, and get the agent to help you with making the site pretty, adding features to the website that you like and make it truly your site. Not a boring site that everyone else has, not for others, but for you.

This was a little bit of a ramble, but it’s really a love letter to my 17 years of note taking. It [[My Note-Taking Path|evolved quite a bit]] sine I started, but stabilized now for quite a while with Markdown and open format with GoHugo.

If you are curious how the notes look, just go to my second brain and check out my public notes and how all my notes are public and connected with an interactive graph and backlinks. You can literally dive into my brain, see what I am thinking about, see how all the notes connect, hopefulyl learn something new for yourself. If you do learn something new, please reach out and write me an email and tell me. I love connecting to like-minded people.

Latest addition to my website here, I interconnected not only my second brain with the graph, but my blog and my book I’m writing at dedp.online, on each chapter, blog or second brain note there is an interactive graph, and you can see links to the second brain and vice versa. Check this note as example, or see [[Future of Blogging]] where I elaborate a bit more on that topic, and included backlinks.

How did you evolve your blogging and note-taking over the years? Any other learnings? I also love the Obsidian Workflow with templates I use everyday and the tools to write in with Obsidian, or check an older article on the same topic Personal Knowledge Management Workflow for a Deeper Life. Check out my other blogs if that interests you. I also share templates in Obsidian, or to clip online articles with [[Obsidian Webclipper]] and its templates.

The whole second brain code is public at my second-brain-public and how I automated the copying of my obsidian notes to my second brain, so I don’t need to duplicate them, is described here [[Public Second Brain with Quartz]].

In Part 1 I argued that the open data platform makes orchestration data and its flow simpler, providing integration into data assets and transformation, ETL, BI and even Kubernetes all while being open, making it a fully composable data stack. This piece continues with what it takes to operate orchestration, and looks at one of the hardest parts of data engineering “DevOps”, and how to manage different environments, business code vs infrastructure code, scalability and how orchestration.

I’d argue that DevOps is what data engineering was a decade ago, when everyone was talking about data science, but the actual work was 80% data engineering. With AI and generative AI, where even more data gets used for model training, or generated by AI, data engineering gets even more critical and larger, therefore also DevOps, which over a certain size gets the biggest task: deploying and upgrading to the latest error-free version, and keeping it stable and scalable.

Deployment, how to Start: DevOps vs. GitOps:

Before you start deploying or operationalizing anything, you will get into [[DevOps]] and more so, [[GitOps]]. That’s why we start with that.

Because once you start with deploying your own infrastructure, you need a way to automate changes across your systems, or even your development server. You can’t manually deploy on different systems. If you only have dev and production, it might work at the start, but there are so many dependencies, stateful NFS mounts where important configs or DWH data is stored, or dependencies between the installed tools from the BI tools to data integration and orchestration.

GitOps is taking that deployment git repo and including infra code such as Kubernetes deployments (also called Infrastructure as Code (IaC)), and making them automatically deployable on git commit pushes to the git repo with post-trigger actions. With the goal of operationalizing data orchestration, we need to look at GitOps and DevOps best practices. GitOps is everywhere these days, with agents working best with Markdown or local YAML or code that is checked in to a git repo.

Orchestration is no different here, we want to have a repo for the code of data pipelines as well as infrastructure and ENVs for each environment (prod, test, dev). We need to separate infra from business-heavy ETL in the data pipeline. We want to standardize the deployment patterns (e.g. base and overlay is how Kustomize is doing it).

How to Operationalize Data Orchestration?

So now that we have GitOps ready and understand it, let’s revise the challenges and then best practices of operationalizing a data orchestration tool. To make it more concrete, I use [[Dagster]] and its features whenever suited.

The Challenges of Deploying Data Orchestration

One important part is how to deploy a system that will be at the heart of your data stack and responsible for the refreshing of your data assets.

The problems are multi-fold. Do you want to auto-scale with Kubernetes, Terraform, meaning you rent machines and scale up and down based on tasks in your data pipelines, spawning up multiple spark jobs, multiple DuckDB processes in parallel as part of spawned containers? Or do you want to pay for it and just connect data pipelines with Databricks Spark Cluster, and hook up Dagster+ that handles it all.

Compare it to the Titanic iceberg collision where too many holds overflowed and then it sunk. Alex Noonan says it’s the same with data at scale, you get too many duplicated data pipelines, tools, inconsistent data contracts, unclear ownership and ungoverned sprawl.

[!note] A handy feature built in: cost observability for cloud
Dagster’s ability to track compute costs per asset, per partition — this feeds directly into your “open data platform” narrative. FinOps for data pipelines.

Best Practices for Running Data Orchestration: Organization vs. Deployment

The problems at scale can get complex and run into a lot of work, quite quickly. Therefore we need a clear setup that works for all our deployments. Here we look at separating the organization-side and the deployment itself.

It all starts once you decide to run your own deployments and machines. Maybe you have a platform team or cloud server that you use with Kubernetes. That’s when we need to separate the state vs stateless, and also the code of data pipelines is a key point here.

1. Organize DevOps means to focus on structure and organization. We need the right organization for structure and repositories. For example, I once created a Kubernetes GitOps deployment Blueprint repo, that includes such structural questions such as GitOps workflow patterns, infrastructure configs, tenant isolation, database migrations, and observability templates for production deployments with Flux.

The organization usually happens in the form of a git repository, and you have one for environment and deployment for infrastructure as code. You set up some sort of GitOps with Flux, ArgoCD, Terraform, or GitHub Actions or similar to automate deployment with a single git commit && git push. All of the environments are typically handled with ENV variables.

2. is Deployment, this is where we handle state and stateless. The deployment repo is usually mostly stateless. It has state for persistent storage locations for configs that Dagster, or BI tools or others need, but these are of a technical nature and usually can be recreated as there is no historic data stored as in a data warehouse where we ingest data from source OLTP databases for example, and version the changes. This way of not breaking state is much harder.

We need to make sure that the data assets produced by the data orchestrator stay intact, and don’t get purged with a new deployment. Other things to take care of are technical resources such as the Spark integration as a resource Python file, which is stateless and can just be redeployed when we update the code, so we have the latest version prepared for the next run.

There’s no complexity, and with proper versioning, we can also easily roll back by just setting the older version in our deployment repo, commit and push. GitOps will then make sure to roll back to a working version. And when the data itself needs repair, partitioned backfills let you re-materialize a specific time range across the whole asset graph, with dependencies resolved automatically.

Code Locations: the Key Piece to Connect State and Stateless

The key piece, again, is connecting state with no state, and Dagster offers an elegant solution here that I suggest for any data work: code locations or also thought of data engineering workspaces. The code locations connect the state and stateless in a meaningful way. Code locations are both stateless (libraries used, technical implementations) but also stateful once the business logic has been run. The outputs are data assets used by downstream processes, and we can’t just delete and recreate them easily. It might also be that some of the assets take hours to create, so just deleting them and then rerunning the full data warehouse for a day isn’t desired.

Code locations are key to a successful orchestration deployment story where we differentiate business from technical implementation logic. Many start deploying Airflow or other data orchestrators and pretty quickly land at the questions: “how do we update data pipelines?”, “do we give people access to the deployment repository?”, “where are all the envs for each environment?”, “how can business users quickly test their Python script or notebook?”, “what about access permissions to the data assets, are they allowed to see all tables, the full S3 bucket?”.

With workspaces you can containerize your data pipeline logic, with its required Python libraries, into a docker image or similar, and test it and deploy independently from the deployment code that is usually only accessible to the platform team or data engineers if they do platform engineering. This leads to a natural separation between data wranglers and data platform people.

Here’s an overview of how centralized teams can structure their code locations differently - from centralized to embedded and to hybrid, with dedicated workspaces for different teams such as data engineering, data scientists, finance or marketing:

The differences are that centralized data team structures can give you consistency across teams, but are bottlenecked and slower when you want to change and iterate. Embedded means that teams have a link to each team, increasing speed and domain depth, but can lead to drift and fewer standards if not careful.

The different setup of teams is also something we have discussed since the dawn of data work. It started with a centralized data warehouse to a more shared Data Mesh and other approaches. With code locations, you can have a centralized data platform orchestrator while still having your domain ETL and data pipelines in dedicated team code repositories, using a hybrid with the best of both worlds.

The code locations also fit into the right abstraction layer as we talked about in the Dagster Almanack, meaning that code locations separate concerns for data teams, and with Dagster resources separate business from technical implementation, that can be used across different code locations (e.g. versioned PyPI packages), making the setup more flexible and adaptable to how data teams work.

Microservices that Work with a Central Open Data Orchestrator

The right abstractions with code locations let you use data pipelines as an advanced microservice with reusable technical code while using sole-purpose domain logic in each location, that integrate into the platform easily with Dagster abstractions and applied functional data engineering best practices. With something like Dagster Pipes, you can invoke external code — Spark jobs, notebooks, R or Rust processes — while logs and metadata still stream back into the asset graph.

If you need a better integration for less technically savvy people, you can use the in-built DSL. DSL stands for Domain Specific Language, which lets you define your pipelines and transformations with a simple YAML interface. Dagster Components takes this further: a toolkit for packaging integrations and common patterns behind a YAML interface, so pipeline instances can be defined with little to no Python.

[!note] General Learning from GitOps
More learnings from using GitOps and deploying orchestration and general data workloads with Kubernetes, here are 8 more points:

1. Separation of concerns is crucial - keeping infrastructure, platform services, and business logic (pipelines) in distinct sections makes maintenance and collaboration easier.
2. Standardized deployment patterns like the base/overlay structure with Kustomize allow for reusable configurations across environments with minimal environment-specific overrides.
3. Versioned artifacts with timestamps (e.g., finance-pipeline-20250512123045.tar.gz) create a reliable release process that enables rollbacks and audit trails.
4. Database migration automation tools, such as Liquibase, can handle schema changes programmatically across environments, thereby reducing manual errors.
5. Test early, test often - validate data pipelines, infrastructure code, and database migrations separately before integration testing in an isolated environment.
6. Workspaces separation from infrastructure code enables domain experts, such as data scientists and analysts, to focus on their core competencies while maintaining deployment standards.
7. CI visibility through lineage diagrams and documented processes enables teams to understand the deployment flow and troubleshoot issues more efficiently.
8. GitOps, as the single source of truth, means changes occur through Git commits, creating an automatic audit trail and enabling pull request reviews.

This overall blueprint, with best practices, helps you specifically with data engineering reference architecture for Kubernetes-based data platforms using GitOps workflow patterns. It includes infrastructure configs, tenant isolation, database migrations, and observability templates for production deployments with linked CI/CD implementation with GitOps infrastructure example.

How to Deploy Dagster: Kubernetes, Helm, Docker

If we go one step deeper, we see different ways of deployment pretty quickly. We will keep this chapter short, as it would fill a full book, but generally you need to decide if you want to run scalable on something like Kubernetes, where pods and compute are automatically scaled up/down, or locally to test, with docker.

Or if you just want to get started without headache and lot of development time, this is where paying for something like Dagster+ service makes sense.

The biggest question for data pipelines is where do you run the compute, do you scale up a Spark cluster, do you run Python or SQL engines as part of Kubernetes pods, or do you use any other way. Because that’s where you will spend most money, as data pipelines are only expensive as the process they run, e.g. crunching large data sets with compute. Partitioned, incremental materialization is the other half of the cost equation. Processing only yesterday’s partition instead of a full refresh is where the orchestrator saves compute money. The same scaling applies to partitioned asset checks, so quality validation only runs on the partition that changed.

You can think about it more high-level, so segregate storage from compute with a Lakehouse or with tables and data stored in Open Table Formats (Iceberg, Delta and Hudi), that saves you money by not using expensive SSDs, but is slower and costs more on the analytics side when you query and join these data.

Here’s a list that helps you get started to operationalize Dagster as the data orchestrator with different deployment options:

1. Helm Charts (recommended for production on Kubernetes): Official chart at helm/dagster: helm repo add dagster https://dagster.io/helm. Two charts: dagster (webserver + daemon infra) and dagster-user-deployments (your code locations), so pipeline code deploys independently of platform upgrades. Guide: Deploying to Kubernetes.
2. Docker Compose (local or single VM, e.g. EC2): one container each for webserver, daemon, and per code location. Example: examples/deploy_docker, guide: Docker deployment. For pure local dev, just dagster dev.
3. Kubernetes customization: there is no official operator — the Helm chart is the K8s story. Runs launch as K8s Jobs via K8sRunLauncher, with autoscaling configured and tunable per asset/job with dagster-k8s/config tags. See Customizing your deployment.
4. AWS ECS / VM (if no K8s): Terraform/ECS pattern, example at examples/deploy_ecs, guide: Deploying to AWS.
5. Hosted: Dagster+: Serverless (they run everything) or Hybrid (control plane hosted, an agent runs compute in your K8s/ECS, deployed via the dagster-cloud-agent Helm chart).

Guardrails and Pitfalls

Beyond deploying and operationalizing the orchestrator itself, there are related topics that cut across data orchestration, some of which we touched on in the open data platform architecture image in Part 1, where you see related tasks we need to deploy when wanting a full open data platform. The most important are testing and governance.

Governance & Testing

A deployment is only as trustworthy as the data it produces. With integrated data quality checks, we enforce assertions on every run instead of testing data after the fact, something a BI tool can’t do, and the source can’t either, because the business transformation hasn’t happened yet. The data pipeline is exactly where the business rules live, so it’s where domain experts can enforce checks and data types they know must hold.

With asset checks, this even extends to assets external to your pipelines. Yes, writing checks is extra work, but it makes the whole system more stable and less complex: numbers are known to be correct, and you avoid the downstream ad-hoc fixes that persist as duplicated data marts or gold-layer (if we use Medallion Layers).

Testing data isn’t comparable to testing software, as data, tools, and volumes change between dev, test, and prod, and upstream from source data sources that we have no control over.

This is again where abstractions make life easier, e.g. Dagster separates business logic from technical implementation in resources. You can swap a mocked or local Postgres resource in dev for Snowflake in prod without touching pipeline code. Combined with the environment separation from our GitOps setup above, functional tests and balancing tests (does the count in the source match the warehouse?) can run in Continuous integration (CI) before a single git push ever reaches production.

Governance goes beyond quality checks: asset-level access controls, audit trails via metadata, column-level lineage for GDPR, and PII redaction in compute logs are the features that make a deployment enterprise-ready.

Obviously, testing and governance are their own domain and field, so that’s worth another article, but I didn’t want to leave it out, as it needs to be part of the strategy and best practices. For example, simply defining what your prefixes for table names in stage, or core, are, or how to name last_updated and created_date across the warehouse and deployment of your data platform or orchestration is key.

Provisioning and Multi-tenancy

Data orchestration is what Dagster is made for, but provisioning is something that fits well in the scope of using Dagster and deployment. As a pipeline rarely runs alone, it also needs an object store, a data catalog, maybe a Spark cluster. Provisioning these accompanying services is itself orchestration, and platform teams increasingly use Dagster for exactly that. It is also an advanced, observable cron for infrastructure. In several organizations, this started with data pipeline deployment, and spread across the company from there.

So without provisioning the needed compute engines or related tooling for your data pipeline, operationalizing the data orchestrator and the full open data platform is not complete. This is also where in larger companies this task usually falls to two different teams, data practitioners (teams of data engineers, analysts, or power users) and the data platform team. The two teams are different, but they have the same need to orchestrate their many tasks and jobs:

For enterprises, this pairs naturally with multi-tenancy and branch deployments: isolated environments that mirror production, so teams can review data pipeline changes like code PRs, and the GitOps workflow from the deployment chapter is applied to the data itself. A concrete example: Lakeflow Jobs can’t express dependencies across Databricks workspace boundaries, but bringing every workspace into a single asset graph can.

Orchestration as the Operating System of Your Data Platform

Closing the initial Dagster Almanack with its guidance, insights and tips for data platform engineers, as with this operationalizing data orchestration guide on DevOps, Infra, and Code Locations in the shared task of deploying a full open data platform.

We have seen how to operationalize data orchestration, and what features Dagster can bring to the table. The best practices are to organize and deploy around git repositories and with separation of concerns in mind. When you think about deployment, always ask yourself where your storage of data assets will be, and how to compute them. This will define the best strategy for finding the best platform for deployment.

The open data platform deployment also helps escape the modern data stack trap. Dagster helps you integrate different tools in line with deployment strategies while separating workspaces for different teams and domain experts in an elegant way that works well with GitOps strategies. Exchanging compute engines and using the best tools for the task is just one config change away, and doesn’t need to change the logic of the data pipeline itself.

With newer features such as state-backend components, Dagster uses these for integrations whose definitions depend on external metadata. Instead of re-querying external systems every time a code location loads, Dagster can fetch that state at controlled times, persist it, and then build definitions from the cached result.

It makes it a composable toolkit for data engineers out of the box and a single pane of glass for the data platform, incrementally adopted across teams and stakeholders. Tools are integrated into a single data platform, with a single deployment strategy, where multiple teams work together.

In the future with more shifting left, DataOps that combines this all under one domain, and with AI agents that govern and help create data pipelines, this foundational data platform is more critical than ever, integrating and scheduling all tasks in a structured and reliable way.

Full article published at Dagster.io - written as part of my services

Creator of Pandas, probably the most widely used data analysis library for Python, Wes has shaped the era of data and is co-creator of Apache Arrow. He also created Ibis to address these issues with a different approach to Python dataframe libraries, by decoupling the dataframe API from the backend implementation.

The article is structured in four parts: (1) how to trust the outcome, (2) knowing what not to build, factoring in cost-per-token among others, (3) accountability of agents and the code they generate, and (4) philosophizing about the future of agentic engineering.

Introducing the Guest: #3 Wes McKinney

Besides creating the most popular dataframe libraries used by most data people, Wes McKinney now focuses full time on agentic engineering with his newly founded company Kenn Software, which focuses on the promise of building a new stack of development and knowledge systems for the agentic era. He’s also doing AI and Python at Posit, where they work on a data science IDE. He’s a part-time investor in various startups.

Wes has been running Claude Code, Codex, and Gemini CLI for months. Thousands of sessions, hundreds of thousands of messages. He has released multiple tools that help the agentic work (more on this later), and he is at the forefront of what’s going on with his recent blog posts about “Why he uses programming languages built for agents, not humans” and Mythical Agent Month, with his recent insights into how to work with agents. Find all his takes at Wes McKinney.com.

I had the pleasure of asking Wes more about these topics, and we’ll go into more details, plus many other things. Let’s get started.

How to Trust the Outcome?

We started the interview with a critical question that stands above all others in the current AI landscape, and I asked him: “Can we trust the outcome?”. What if we need something important, other than a hobby project? What if the data must be correct (hospitals, banks)?

Similar to what Mark Freeman told us in our last interview about using spec-driven development with spec-kit, Wes uses a similar approach, but with an agentic skill framework called superpowers (currently 216k stars on GitHub). Compared to spec-kit, it specs out the requirements differently by (A) guiding you through the conversation, asking you the right questions to get to what you want to build, and (B) once you fire it off, it spawns a sub-agent that keeps the implementing agent on track. Wes said, “Superpowers looks for drift”, and course-corrects if the implementing agents drift off to non-relevant, or not even specified, tasks.

Wes spends a lot of time in this specification phase, sometimes hours, very detail-oriented and engaged. Even before he starts speccing, he has subconsciously worked over the topic and idea for a long while. He will not start implementing something when he doesn’t know super clearly how it fits together. The insights, the architecture, come from him. But the interview style by superpowers helps him clarify his thinking.

He doesn’t only give his feedback to the questions, but sometimes also fires up multiple agents and integrates their feedback. Codex models especially seem to work well for design questions.

He puts a lot of importance on the spec being:

1. Spec conformant: Meaning the agents act in accordance with your specific set of rules, standards, or specifications.
2. Code correctness and quality: This is where Wes uses e.g. Roborev, his own created AI-reviewer.

Correctness is crucial, which led to creating Roborev. Wes developed many tools that help him work agentically, and we’ll hear about many more later. Roborev, for example, is a code reviewer that can be initialized with a hook on a git repository, and from that moment on, every commit will be auto-reviewed by Codex (the default, but you can choose others too).

I use Roborev myself, and this is what the interactive TUI looks like - showing the most recently fired hooks with their running status, but most importantly, whether the review passed (P) or failed (F):

If it failed, you can open the review and see detailed findings categorized into severity low, medium and high:

The convenient workflow is that you copy the review with y and feed it back to your running agent to let it fix things directly. The current agent that created the change works best, as it already has all the context, compared to starting a new one that needs to load context and what has been done.

Roborev also helps to review a smaller part at a time. Wes also says it will never catch all the errors, but LLMs are very good at pattern matching, which is what error finding is, and they find many that might be missed. On top, he adds reviewers with different roles, e.g. giving agents roles such as focusing on security, CI, software development, or performance, which gives much more accurate feedback than a general reviewer.

After having gone through the spec intensively, having made sure that drift happens as little as possible, and having auto-reviewed each commit by Roborev, what is left for him to review is much less now, and of high quality. He then reviews the code and checks that it looks and does what he expects or envisioned.

Wes has a very clear problem or idea that he then solves meticulously. However, at the same time, he runs agents in parallel and works on many projects concurrently, context-switching between them¹.

[!note] Rigorous process in place needed: Changing models

The rigorous process he does is also needed because the models are constantly changing and are very unpredictable. It is hard to have a consistent outcome if you do not have reviewers and processes in place. And Wes says these AI reviewers are much better than just static analysis.

How to Maintain Agentic, or General Projects over Time?

The second question was about maintaining projects and how Wes handles maintenance, as creating projects is usually the easy part, but maintaining them for years to come is difficult. And how does he see that in combination with AI? Will that be outsourced to AI?

First of all, Wes uses his own projects and tools. That’s the reason they exist, and it helps him find bugs. This is why he fixes errors or bugs when he runs into them. Besides Roborev, which helps tremendously to review and have fewer errors while developing, he uses Middleman to keep an eye on his agents and projects. It’s another tool he built that gives him a local-first GitHub dashboard and triages what to maintain or fix from other users.

He automated repetitive work such as releasing with a full release script so he can release fast and fix bugs fast. The Changelog on GitHub is fully streamlined, too. He is also careful about what comes into the main branch, only changes he has verified and assessed as “pass”.

To illustrate what Wes is maintaining, here are some of the projects Wes built recently, some of which he might not have built without AI:

• roborev: Continuous code review for AI coding agents. Runs in the background and surfaces issues per commit before they compound.
• middleman: Local-first GitHub dashboard for maintainers to triage, review, and merge PRs and issues across repos.
• agentsview: Local coding agent session viewer for Claude, Codex, and Gemini with analytics and full-text search.
• msgvault: Archive a lifetime of email and chat locally, with full Gmail backup, DuckDB analytics, a TUI, and an MCP server for AI queries.
• moneyflow: Personal finance data interface for power users, supporting backends like Monarch Money and YNAB.
• Spicy Takes: LLM-analyzed blog posts from 20+ prolific tech writers, each with a TL;DR, key quotes, and a spiciness rating.
• VibePulse: Simple macOS menubar app to monitor Claude Code and Codex token consumption.
• kata: Local-first issue tracker for AI-assisted software work, with an agent-friendly CLI and human-facing TUI.

[!note] Earlier Tools and Frameworks Wes Has Built

• Positron: A next-generation data science IDE built on VS Code, supporting Python and R.
• pandas: The most widely used data analysis library in Python.
• Apache Arrow: Language-independent columnar memory format for analytics.
• Ibis: Portable Python dataframe API that works across any backend.

Building for Maintainability: Modular?

I asked him if he builds for better maintainability, e.g. builds in a modular way so the AI agents can easily fix something or create a feature in a dedicated area without breaking the full program.

He didn’t answer the modularity part directly, but Wes implements and uses tests extensively. If something needs to exist, he writes a test for it. But even more, by investing in test infrastructure, regression tests help prevent bugs and protect existing features during rapid development.

He also mentions that bugs are created faster these days, but also fixed faster.

How to Decide what to Build? Saying No!

Given that AI can get addictive, and in a time when you can build almost anything, I asked Wes how he knows what to build, and when to say no to avoid building the “wrong things”.

He said that:

It’s not the ideas on their own, he’s thinking a lot about what he wants to build.

Again, it is in his subconscious. He thinks and asks himself all day: “How is it beneficial for agents? For humans? How can it be applied?”

If he can’t explain it, he will think more. For example, msgvault didn’t have a web interface, and he could have easily added one from the very beginning, but he didn’t have a clear picture. So he just postponed it until later, when he had a use case, a pain point, or a real need.

'Those are the constraints', Wes adds. 'Because if you don't, AI will bring in lots of crap'.

Superpowers also helps him with guardrails by keeping the AI on track. Besides, Wes has a perfectionist mindset, making him want to perfect the tool that works for him and improve the workflow.

When He Was Building without AI: Pandas

It was the same when he was building Pandas: he was building it for his use case when fiddling with Excel. Then there is taste.

Every prompt, every decision in the spec phase adds up to 100s or 1000s of small decisions, essentially manifesting one’s taste. That’s why the product comes out differently from two people, even though they use the same LLM models.

[!note] Find more at AI Council Talk about the Scope, Design and Taste in the Mythical Agent Month

Wes gave a very insightful talk at AI Council 2026 about this very topic, called the Mythical Agent month. He said what is left is “Scope, Design and Taste” with Conceptual Integrity (from the book by Fred Brooks).

Saying No is Our Last Defense

In his recent slides, he shares “When code is free, saying no is our last defense”:

Every new feature is cheap to create but expensive to maintain. Each one adds surface areas for bugs, confusion, and future agent mistakes.

[!tip] “Hell Yeah or No”: A similar term by Derek Sivers
Similar to Wes’s figuring out and saying no as our last defense, Derek Sivers said something similar before, where you say no to everything until you feel “Hell Yeah”. This Hell Yeah or No approach doesn’t seem to have changed much with AI. This doesn’t apply only to AI, but also to life and career, in my opinion.

Cost-per-Token at True Price Will Stop the Waste

A very current topic is how the growing cost-per-token factors into this decision of what to build. Or does it not? There’s even a term called token maxxing that encourages programmers to use more tokens, whether by the company or by peer pressure on X/Twitter.

Wes was at the top of the HN leaderboard at some point, currently on #4:

Wes’s current usage is ~$20,000/month at API rates, which he sees on another tool he built called AgentsView. He said that

He thinks that all his high-quality output through the shared tools or the work he does is higher than the invested money.

But on the economics side, he thinks that:

Subscriptions go away, and pay by usage, a good thing. AI slop and low-value projects go away. This helps pay the true cost of tokens, which isn’t the case for now, making the consumption (or even waste) of lots of tokens non-problematic.

Enterprise Token per Employee: Clarify Useful vs. Vanity AI Work

This was actually one reason why he built AgentsView: to have an overview of your own usage, a better “token intelligence”, but also at a larger company to measure each developer’s usage. It could be part of performance reviews, showing each user’s token spend vs the value generated.

You’d have to justify your tokens, the opposite of now, where developers at Meta or Amazon are expected to burn tokens without incentives. Right now, it’s the wild-wild-west (something previous interview guest Chris Riccomini also said).

Accountability of Agent-generated Code? Who is Responsible?

My next question was how do we make people accountable for things they didn’t create (vibe coded)? I gave the example of self-driving cars: who takes accountability if a Tesla hurts someone? (That’s one reason full self-driving is still not allowed in Europe, as it’s legally not settled who is accountable.)

Vibe Coding ≠ Vibe Coding: But Agentic Engineering

Wes made clear that what he does is not vibe coding, but agentic engineering. All the planning and architecting with superpowers and his newly created tools is not the same as vibe coding.

The term vibe coding to him means when you just one-prompt it, don’t look at the code, and ship it. Again, this is not what he does.

He says:

We can’t disengage from planning and writing specs. We can move much faster, but don’t vibe code. Vibe coding is very dangerous and irresponsible.

Like the Coinbase example, he finds letting non-technical employees push to production highly dangerous. We humans, with fundamental understanding and seniority, need to be more engaged in designing and testing, as coding is essentially “cheap” now.

He continues:

Automated code review certainly helps, but it isn’t a substitute for engineering experience.

Philosophize about the Future with Agentic Engineering

Wes is also an investor, a person who foresees the landscape well with his involvement in major data libraries. I asked him: “If you think about AI, where would you invest your money? What do you trust will have the most benefit or will work well with AI?”

Where do you see the future heading, or where does this end? Especially when we talk about data engineering?

Future of Data Engineering

He says that he is not involved too much in data engineering anymore, but that he is an investor in dlt, MotherDuck, and Bruin. But his main focus is on agentic work, somewhat on top of the “dbt legacy”².

But what he sees as currently the hot topic is Headless BI, custom dashboards, and building a semantic layer for better context for agents. Things like business rules and sending the “right” queries. Building new knowledge systems for companies. For example, through msgvault, which extracts value from years of emails and easily makes them searchable.

He saw people building personal CRMs on top of msgvault and their emails. That’s the current direction we are heading, he says.

How Do We Still Learn? By Learning by Osmosis

The challenge will be: how do we develop senior engineers without writing code anymore? Wes himself doesn’t write much code anymore, but reviews, guides, and adds taste. I asked him how someone can gain the work experience he has without the coding or going through the pain of coding, while avoiding the danger of not learning anything new, or getting overwhelmed with constant stimulation and potentially becoming addicted.

He says the hard labour goes away, which is where we usually learn. This is the way of learning by osmosis³, where we acquire knowledge while failing or naturally through exposure and immersion. He thinks the focus needs to be on design patterns and understanding architecture, to have the technical vocabulary to guide or understand the agents.

Next Interview

I hope you enjoyed this interview number 3 with Wes. Huge thanks to Wes for taking the time to speak with me and for sharing his experience with all of us. Follow him on Website, LinkedIn, X/Twitter, or on Bluesky, and follow along on his new company Kenn Software, or check out his agentic engineered tools he built at GitHub.

There is one more interview already lined up with none other than Maxime Beauchemin, so please share feedback, questions you might want to ask, or just your experience on how to work with AI in the data space. We’re all in this together, figuring it all out. The more we can learn from each other, what’s important, and maybe also what’s not, the better.

Full article published at MotherDuck.com - written as part of my services

So how do we make data agents reliable and accurate for database queries? Everyone talks about harnesses, evals, and context layers, but the real challenge is bringing them together with data engineering and the context you already have, such as database schemas, a semantic layer, metric definitions, plus the business knowledge that normally never reaches the agent.

That’s the question this blog tackles: how agents can work with the data stack and analytics, and how a context layer fits in. We also take an inside look at ktx, a new context layer that reads from the usual sources but also the less obvious ones (Markdown, Notion, etc.), driven by agentic workers.

The idea is to pull two kinds of knowledge into one reviewable place: the hard semantics (your warehouse schema, joins, and metric definitions as YAML and SQL) and the soft semantics (the business context living in docs, wikis, and Notion that agents usually never see). Both are committed to git and reviewed like code, so a human stays in the loop while agents get a warm start instead of a cold database connection. The payoff: more accurate answers with fewer (and cheaper) queries against the warehouse.

SOURCESWarehouseschema · metricsBI toolsdashboards · joinsNotion + wikisoft semanticsDocs + Markdownnotes · contextCONTEXT LAYERAuto-built. Reviewed in git.Hard semanticsYAML + SQL the warehouse runsSoft semanticsMarkdown the team readsreviewed like codeAI agentaccurate SQL · governed

A context layer ingests your warehouse, BI tools, and docs into hard and soft semantics, then serves governed SQL to your agent.

Modeling with Analytics AI Agents, with a Context Layer

Every business or data analyst faces mediocre results when prompting Claude or Codex on their data stack. It might figure out some context by reading the git repos or other metadata it can find. Still, the hard part is teaching the internals and business context that are unique to each business domain and company. So, how do we make AI agents for data and analytics reliable and accurate for database queries? Do we need to manually copy and paste Google Docs and Markdown files into the prompts, or can a context layer provide a more reliable, safe, and governed way to ask an AI assistant for analytics?

These days, it’s much easier to ingest or add almost any number of new data sources with custom-built ETL data pipelines, whereas before we had to make hard decisions about what to include and what not. An AI agent, such as Claude or Codex, can merge multiple data pipelines that access the source database via CLIs and destinations via MCP, API, or CLI. But we still need an API and a process for updating source data, not just once. We need to make sure, test, and verify that the data is correct, potentially more than ever.

The challenge remains in modeling the data in a way that represents what the business is, making sure data flows fast but is also correct. But any AI assistant is only as good as the context we give it, and how easily it can read and express context, metrics, models, and knowledge. So, does the context layer solve these problems?

The context layer primarily supports the accuracy of SQL queries, continuous updates to the business context, and governance. Additionally, with a newer context layer, we can include more relevant business insights that are stored internally, usually in unstructured form, in tools like Notion as business documentation. Traditionally, these data weren’t included because they weren’t a “pure data source”, but they can be helpful for decision-making, especially when done in an agentic way. Every bit of human-written data can help the agent make better decisions. So the ultimate problem we solve is to include more useful data and have a more agentic engineered way that is faster and more streamlined, especially to start with.

What’s an Agentic Context Layer?

The core is to turn warehouse metadata, BI tool definitions, query history, docs, and approved metric definitions into reviewable files that agents can search and execute. We feed valuable metadata and actual data to agents and humans so that they can be ingested and discovered more easily in one place.

It’s a further evolution of a [[semantic layer]], such as Cube or AtScale, which are more focused on the modeling of metrics and domain knowledge and usually have less extensive knowledge of documentation, metadata outside of metrics, joins, and source data schema. And building a semantic layer from scratch can take time, as we need to extract metrics and unify them in a single repository in an additional layer. The main driver for a context layer is that it’s automatically generated and includes additional business knowledge that lives outside the technical containers of data engineers, such as data catalogs, DDLs, or YAMLs, in tools like Confluence, Notion, and internal wikis.

I have written about how to model data in the agentic era, Data Modeling for the Agentic Era (Semantics, Speed, and Stewardship), and we’ve seen how metrics and context can strengthen a BI tool or insights. The context layer is all about context that we already have from our database catalogs (information_schema, data types, tables), more technical metadata, but also from our metrics inside the BI tools or in a declarative YAML, while adding new sources such as documentation in Markdown, rich text, fetched via API, CLI, etc. All with the goal of giving agents and LLMs more information to make the right calls autonomously.

Maxime Beauchemin calls documentation, wikis, and Markdown soft semantics, whereas YAML and the SQL metrics are hard facts or hard semantics. More soft semantics, such as business context from domain experts, help agents, and we need to make them available to other agents and humans too, building a full context layer that grows fast, where governance is essential.

Enhancing Gen AI Trustworthiness

Google describes How Looker’s semantic layer enhances gen AI trustworthiness, and a semantic layer can reduce errors for generative AI and autonomous agents creating queries. They say:

With the foundation of a semantic layer, rather than asking an LLM to write SQL code against raw tables with ambiguous field names (e.g., order.sales_sku_price_US), the LLM is empowered to do what it excels at: searching through clearly defined business objects within LookML (e.g., Orders > Total Revenue). These objects can include metadata and human-friendly descriptions (e.g., “The sum of transaction amounts or total sales price”).

We also acknowledge that data modeling for agents is not the same as data modeling for humans, and that shapes how we work with agents and context together.

Context Layers Saving Literal Money

In the end, it’s also saving money, as more governance means the agent knows the system in more detail and therefore needs to make fewer queries to the actual database, or can do so in fewer iterations.

Queries on Snowflake, BigQuery, and Fabric can be expensive, and agents work at a different speed than humans. If we add more context to the agents, they need fewer queries because the information can be extracted from the retrieved context.

The agents have access to all this data, either directly or through CLI, skills, or MCP. The big advantage is that the agents get a warm-up and do not start with a cold database connection. All the warehouse schema, business semantics, BI usage patterns, and human documentation are in one single place.

Also, on the other end, verifying that the queries are correct is hard. More context can help with this.

Entering ktx: Open-source Context Layer

Kaelio launches its new open-source context layer solution called ktx. The goal is to get an OSS context layer solution that everyone can use for the future of analytics.

It started as an AI data agent platform, with the agent performing well on accuracy benchmarks against other platforms, which helped make the case for developing it further. Dashboards and agent interfaces are easier to build than ever, but these data apps need more durable and complete knowledge from all the company’s data. This extended data added with ktx makes the agent more accurate.

[!info] Want to get started immediately?
You can quickstart with a demo project that fetches data from Postgres and Notion, with a dbt project and a Metabase interface. See Quick Start with public demo data, or check out the GitHub repo for the full code. If you already have a project with relevant context and sources, get started with a simple prompt: Run npx skills add Kaelio/ktx --skill ktx and use the ktx skill to install and configure ktx. More at the Quickstart docs.

How Ingesting Business Context Works

So how do we enhance business context for agents to make use of it?

Ingestion happens through two parallel items:

• ingests and prepares both data context such as data models, historical queries, BI dashboards, etc. and
• business context such as internal docs, Slack, etc., in a format that data agents can use to operate more reliably and with stronger governance.

If we look at a high-level overview, compared to a common semantic layer or logical layers in BI tools, you can build a knowledge layer around your business, with not only structured data but also unstructured knowledge from your company-wide internal documentation on Notion, or your data in Markdown or a git repository. ktx helps navigate and maintain this business knowledge.

Both of these can substantially enhance your “context”, basically critical business insights that are usually internal or in a format that can’t be easily used. The challenge here is obviously to separate the updated and accurate documentation or knowledge from the outdated, which needs more focus or rigid process flow/rules on documentation to keep the latest updates, adding an owner and also going back to docs and invalidating them as no longer relevant to improve the context quality.

A knowledge layer wraps structured data and unstructured company knowledge into one place agents can use. See the full video.

Smart agents such as Claude, Codex or any other that get context have built-in tools to understand business metrics, and combined with this newly ingested unstructured knowledge, they can help us get accurate responses against our warehouse or across our various data sources, and act as the human curator of data sets and entities for BI dashboards and apps.

Agents pair their built-in reasoning with ingested unstructured knowledge to answer questions across your sources. See the full video.

If you compare using raw Claude Code vs. the ktx semantic layer component¹ with extended context, you get faster and more correct answers with ktx. Here’s such an example:

ktx has basically two connected sides where the first one builds and maintains the context layer, and the second serves that context to agents at runtime, which you see above.

Anatomy of Context Layer in ktx

The anatomy of a context layer is based on two files and two jobs. YAML for what the warehouse can execute. Markdown for what the team needs to interpret it. Both are committed to git and reviewed like code.

The wiki is the self-organizing collection of ingested Markdown files. And for the YAML files, ktx has a similarly self-organizing executable semantic layer.

ktx ingest pulls raw data from your data stack into a context layer, where you can also edit and create additional context manually. Additionally, you plug ktx into your agent of choice, which allows that agent to send extra memories for ingestion to keep the context up to date based on actual source data.

The semantic part is a set of functions that define metrics precisely, like “monthly revenue” or “profit”, so if you ask your agents, ktx knows exactly how to pull the data and answer the query with SQL, running it consistently.

Context as Code: How to Use Context Layer

ktx is open source and you can install it with npm install -g @kaelio/ktx. It has a self-improving context layer that teaches agents how to query your warehouse accurately, from approved metric definitions, joinable columns, and business knowledge it builds and maintains for you.

With the ktx CLI you get context as code, writing wiki pages and semantic-layer definitions as git-based files you can review, diff, and merge. This is to update and enhance current context (documentation) you have in your company, and provide it to your agents as well. Git-based, so humans and agents can collaborate on it by editing the same context.

So how does it work? Here are the most common commands you can run and use. We don’t go through the whole CLI reference, but cover the most important ones to get a feel for how to work with the ktx CLI. Six commands cover the loop from check status → discover → verify → serve.

[!note] Setup: Demo data
This assumes you have set up ktx and want to work with it. If you want to follow along, or build up your own, check the quickstart with public data Kaelio provides.

1. Is Everything Wired Up? → ktx status

First, get the status, which tells you which connections, LLM, and embeddings are live before you do anything else.

1

ktx status

1
2
3
4
5
6
7
8
9

  LLM           claude-code · sonnet  ✓
  Embeddings    all-MiniLM-L6-v2 (384d)  ✓
  Storage       sqlite (state) · sqlite-fts5 (search)

  Connections (4)
    ✓ orbit      postgres
    ✓ metabase   metabase
    ✓ dbt        dbt
    ✓ notion     notion

2. Discover & Search Metrics → ktx sl

sl is the semantic layer, the YAML definitions that tell agents how to turn “revenue” into correct SQL. Bare ktx sl lists every source and you can add a query to search. Add --json when an agent (not a human) is reading.

1
2
3

ktx sl                       # list all semantic sources
ktx sl "revenue" --json      # search, machine-readable
ktx sl "int_active_contract_arr" --json

The JSON result is rich: a relevance score, a snippet, and matchReasons showing why it matched (lexical, token, semantic). Trimmed:

1
2
3
4
5
6
7

{
  "name": "int_active_contract_arr",
  "path": "semantic-layer/orbit/_schema/orbit_analytics.yaml#int_active_contract_arr",
  "columnCount": 5,
  "score": 0.036,
  "matchReasons": ["lexical", "token"]
}

This is what makes an agent cheaper and more accurate as it finds the right metric from context instead of guessing field names against your warehouse.

3. Verify Before You Trust → validate + query --format sql

When you edit a metric, always test first against the live schema with the following command:

1

ktx sl validate mart_revenue_daily --connection-id orbit

validate catches missing columns, bad joins, and unsafe source names, so you fix the YAML before it hits production data. Then compile the SQL and read it before running anything:

1
2
3
4
5

ktx sl query \
  --connection-id orbit \
  --measure mart_revenue_daily.total_net_revenue \
  --dimension mart_revenue_daily.revenue_date \
  --format sql

Swap --format sql → --execute --max-rows 100 once the SQL looks right. The pattern is always: compile, eyeball, then run.

4. Search the Soft Semantics → ktx wiki

Metrics are the hard facts. The wiki is the soft semantics we were talking about, such as the business definitions, rules, and gotchas in Markdown that agents search for context. Same shape as sl: bare lists, query searches.

1

ktx wiki "new hire"

1
2
3
4
5

  GLOBAL (4 pages)
    #1  new-hire-onboarding-requirements  — what every new hire must know by week one
    #2  orbit-arr-methodology             — contract-first ARR definition + NRR treatment
    #3  orbit-company-overview            — what Orbit sells, plan tiers, workflow
    #4  implementation-handoff-process    — Sales Ops → CS handoff requirements

When sqlite-fts5 is configured, wiki search is hybrid with lexical, token, and semantic (embeddings) lanes combined and ranked together. That’s how an agent finds “ARR methodology” when someone asks about “annual revenue.”

5. The Raw Escape Hatch → ktx sql

Need to peek at the actual rows? ktx sql runs read-only SQL (single SELECT/WITH only) against a connection.

1

ktx sql --connection orbit "select count(*) from orbit_analytics.int_procurement_qualifying_actions"

Most of the time you want ktx sl query, not raw ktx sql. Reach for sql only when you genuinely need rows the semantic layer doesn’t model yet.

6. Hand it to Your AI Agent → ktx mcp + ktx setup

This is what your business users or engineers might use most often, connected with actual agents. Start the MCP server, then wire up your agent of choice:

1
2

ktx mcp start
ktx setup --agents --target claude-code

setup drops a .mcp.json into your project and installs the analytics skill. Next time Claude Code opens in that directory, it picks up the ktx MCP server and starts answering questions with your governed context, calling ktx sl and ktx wiki under the hood instead of hallucinating field names.

1
2
3
4
5
6
7
8

● Skill(ktx-analytics)
  Called ktx 8 times

● Here are the metrics defined in the orbit semantic layer:
  mart_arr_daily         arr                  ARR ($) — contract-first
  mart_revenue_daily     total_net_revenue    gross − credits − refunds
  mart_customer_health   at_risk_customer_count   distinct high-risk accounts
  ...

The agent discovered every metric on its own. No copy-pasted schema, no guesswork.

The Whole Loop

Recapping the full workflow in one loop:

1
2
3
4
5

ktx status              # 1. is it wired up?
ktx sl "revenue"        # 2. discover metrics
ktx sl validate …       # 3. verify against schema
ktx wiki "ARR"          # 4. pull in business context
ktx mcp start           # 5. serve it to your agent

When you’re done, shut the runtime down cleanly:

1

ktx admin runtime stop

Go deeper: Building Context · Writing Context · Reviewing Context · Serving Agents · LLM configuration

The Workflow: From Source to Context Layer

From reading configured source connectors (1) from databases, BI tools, modeling code, docs and notes, to context building and extracting the context (2) of each, to reconciliation (3) of creating new semantics and metrics based on it, to validation (4) and checking references before agents rely on them.

Resulting in a wiki and semantic layer that is referenced and self-improving, and this is how the semantic layer part looks within ktx:

Why a Governed Context Layer Matters

Wrapping up, I know this has been a lot of new information, but I’m very excited about the direction of ktx and integrating logical layers such as metrics and semantic layers with more advanced business context, and with everything being fully open source for you to try out.

But the challenge of natural language for semantic layers being too imprecise is real. A context layer with hard and soft defined configurations will help the conversational interface via the agents tremendously. More context doesn’t always help, but if it’s well maintained and fetched from the actual source or pulling in actual domain knowledge from Notion pages, it’s priceless.

Also, the feature of easily automating commands via CLI and adding or fixing wrong context in a central repository quickly is super helpful. This keeps the human in the loop and provides a consistent way to verify a governed layer that holds the company’s context while using the power of agentic engineering.

If you like this, try ktx with the getting started guide, and star it on GitHub. It’s all open source.

Full article published at Kaelio.com - written as part of my services

My goal is to offer a collection of wisdom, insights, and principles gathered over the years. Giving you an outside view from someone who has used Dagster since back in 2019, used it at enterprise scale but also for my hobby projects (e.g. real-estate project). The piece should give you a holistic view of Dagster’s place in the data ecosystem, how to deal with the complexity of data architecture and enterprises, and scaling your data jobs.

This article shows you how orchestrators such as Dagster are built for an open data platform that integrates the full data ecosystem, with the shift to data assets instead of DAGs, reducing complexity and applying data engineering best practices.

[!note] Definition of Almanack (also spelled “almanac”)
The term refers to a publication containing a variety of information on a dedicated topic. The modern usage of Almanack, particularly in the context of books like those by Charlie or Naval Ravikant, is often metaphorical. It suggests a collection of wisdom, insights, or principles gathered over time.

What is Dagster

In late 2018, on a co-working and co-living sabbatical in Bali, I was searching for something to bring the data warehouse out of the drag-and-drop world of SSIS and Oracle reporting and into a code-first, developer-friendly workflow. I looked at ODE, BiGenius, TimeXtender, and WhereScape, but found that none of them quite fit my open source and programmatic preferences, so I tried to build something myself but didn’t succeed. A year later, back at my 9-to-5 in Copenhagen, I heard Nick Schrock on the Data Engineering Podcast describing the motivation and story behind a Python framework called Dagster that did exactly that. I was hooked, and have used Dagster ever since.

Early Focus on Developer Friendliness

To understand the context of 2019, you must understand that back then, most ETL jobs were triggered with cron or bash scripts, and if there was an error, the only option was to re-run in the next nightly window where production wasn’t touched. Dagster, as explained by Nick in the podcast, focused on developer-friendliness, in particular for ETL developers back then, and that focus hasn’t changed today for data engineers.

So what is Dagster? The original idea, started in 2018 during a sabbatical after Nick worked at Facebook, came with this definition:

One of the goals of Dagster has been to provide a tool that removes the barrier between pipeline development and pipeline operation, but during this journey, he came to link the world of data processing with business processes.

Today the definition hasn’t changed much and reads like this from the Dagster Docs:

Dagster is a data orchestrator built for data engineers, with integrated lineage, observability, a declarative programming model, and best-in-class testability.

The initial definition to “link data processing with business” was the key reason that brought me to it, along with the quality of how the components were implemented. Even more compelling was Nick’s visionary outline for 3-5 years ahead: to make the work of data engineers similar to software engineers, and make their daily life easier.

Biggest Shift Early On

This vision led to many new concepts Dagster originally created, which we take for granted in today’s data work, and shifted the work into a more reliable and useful toolset for data engineers.

Data-aware Orchestration Shift

One of the biggest shifts compared to previous tools and orchestrators was that orchestration was fully data-aware from the very beginning. It tried to understand the heterogeneous complexity that exists at every small to large enterprise company, and thrive in it, supporting the full data engineering lifecycle with its platform and data pipeline capabilities built in.

This gave me a toolkit for building reliable data pipelines out of the box early on, with battle-tested features through its users (open-source) and a quality and thoughtfulness I hadn’t seen before. This was personified by Nick and could be vividly felt in the early interviews, but also in the code that the team produced openly on the repo.

For example, backfilling, restartability, or Spark integrations were open on GitHub, to use, and to adapt to your needs. It was a bit like dbt, but instead of modeling your SQL queries, you’d model your data pipelines and integrate complex data architecture.

This also meant that moving datasets or integrating dependencies is strongly supported, not an afterthought. In Dagster you can use resources to work with Polars, Pandas, Arrow, DuckDB, or anything else to pass datasets, and reference data assets declaratively, or even non-existing ones as Dagster knows to create the assets. Compared to Airflow, where you could load only small data with XCom for the longest time¹, this makes code simpler to understand and maintain.

Resources also decouple storage from compute. You could use Apache Spark locally with a single JAR file and in test use a full-blown cluster, or use MinIO as an S3 interface with simple bucket configuration and in production an S3 server from Amazon. Both are interchangeable without changing pipeline logic, by pure configuration — that’s the beauty of declarative data systems, and Dagster embraces this to this day.

Shift to Data-Asset Based Orchestration

Dagster was super early in shifting from DAGs and operational task-based orchestration to asset-based ones such as BI dashboards, tables, reports, ML models — artifacts the user actually cared about. Everything shifted from an imperative approach to a declarative one like Kubernetes or React, where you define what the dataset must or should have, and Dagster takes care of the implementation logic, mapping it to the configured filesystem, compute engine or cluster.

Now you could quickly describe each pipeline with declarative notations like update: daily or change to update: monthly, or if more advanced, you could define a small sensor logic that checks S3 for updates. Instead of implementing all logic in a data pipeline, we apply the logic directly to the asset and closer to the data, which makes it more transparent and integrated into the full data lineage. When updates happen, we have the full graph, but also a leaner and easier-to-maintain setup.

Or zoomed out - you see the focus on the assets, the tables themselves:

This led later to “Software-Defined Asset” and its mental model where you can define an asset pre-runtime and declare connections to upstream or downstream data assets (e.g. real-time housing prices that we fetch from a webpage that does not exist beforehand). Now we can already build and implement our graph and data lineage without having to physically create the dataset first. Software-defined assets use code to define the data assets and are version-controlled through git and inspectable via tooling. This transparency allows anyone in your organization to understand the canonical set of data assets and reproduce them at any time, and also lays the groundwork for asset-based orchestration.

As Rich Hickey said, the aesthetics of a programming language do not matter, only the outcome. Assets play into that fact. In data engineering terms, it’s not code but data pipelines and DAGs, but what everyone cares about are their outcomes: the data assets.

That led to the shift from working with tasks and DAGs to data assets. That’s the developer-friendliness built in from day one: develop locally and deploy to test and production, with infrastructure and technical implementation decoupled from business logic.

It made Dagster a bit more complex to start with — you need to know more upfront — but since every enterprise hits these data engineering challenges eventually, it’s better to embrace the fact and build for it. The result was an improved developer velocity, but what I noticed, too, was the joy of building reliable data pipelines, equipped with tools that helped me deal with errors, infrastructure, multi-tenancy, data science, big data, and everything thrown at me back then.

Not only single Purpose

When I first introduced Dagster at my previous company, all of a sudden other teams started to take notice and also wanted Dagster for other work such as provisioning infrastructure with one-click deployment. Especially the cloud platform team needed a tool to automate its scripts to deploy on Kubernetes, OpenShift, and everywhere else, but other teams also had needs to automate. With Python as the programming language for Dagster, reading from an external FTP server, transforming the data, and uploading it somewhere via API were not multi-month projects across different teams. Dagster’s flexibility and integration into other tools and systems were a key strength for most teams.

Observability and monitoring were another addition. Every run was logged in the UI and everyone could see the rich metadata of each pipeline run. And because it was open source and had a rapidly growing community, support and ideas didn’t run out.

Dealing with the Complexity of Enterprise Systems

If you have worked at any company larger than 10 people, you have noticed pretty fast that you are dealing with multiple source systems, different CRMs, different ERPs, multiple cloud platforms. Most enterprises have all major cloud platforms running in production, whether it is Amazon services, Google GCP, Azure, or any other major platform. You as the data engineer are the one making sure to integrate them, and basically deal with the complexity that comes with it.

How to Reduce Complexity?

First, acknowledge it: heterogeneous data complexity is a fact of the enterprise data lifecycle. Second, lean on tooling with technical integrations and written code that implements each vendor’s API, so we don’t build everything from scratch repeatedly. Third, work around the actual data assets the users want, not DAGs. With assets we declare outcomes, tests, and dependencies, and the system handles the rest. That cuts dependency hell, an unproductive Software Development Lifecycle, and the fear of change.

Composable is Making Systems Simpler

A more holistic framing comes from Simple Made Easy by Rich Hickey, creator of the Clojure functional programming language, where he debates what makes systems complex: state and objects, lots of vars, syntax, inconsistency. His conclusion is that composable is what makes systems simpler (like in music for a composer, which is what he created Clojure for): the ability to assemble, reassemble, and swap individual components into a flexible whole.

Dagster has exactly that ability, too. It integrates the core principles of Functional Data Engineering — pure and idempotent tasks, immutable partitions, reproducibility, versioning — directly into the framework.

State, Hickey points out, is never simple. Unfortunately for us, data engineering is all state: every datum is tied to a timestamp of when it was created, processed, or backfilled. Fortunately, helping us manage that state is what Dagster does: data assets, virtual assets, partitions, incremental materialization, and partitioned asset checks that evaluate a specific partition of an upstream asset instead of the whole dataset.

Rich concludes “Simplicity is a choice”, echoing Leonardo Da Vinci:

Simplicity is the ultimate sophistication

(Open) Data Platform

But how did the shift and the data engineering principles evolve, and how can we apply these as a unified solution?

With Dagster’s data-aware orchestration, shift to assets, separation of concerns, and multi-use, we are automating harder data and infrastructure deployment problems, and Dagster solves the problem of managing complex data environments more holistically. To me, it feels as though Dagster gave me this peace of mind and the toolset to simplify data engineering in a complex environment early on, and is the right tool for the ultimate sophistication for data work.

Dagster’s approach is composing a data orchestrator that integrates into any type of data work: from data integration with dlt, to transformation with dbt or just Python logic, to updating BI dashboards, to deploying on Kubernetes, all into a unified system. A fully open data platform, making orchestrating data and its flow simpler.

All of these features, combined with DevOps deployment strategies, make Dagster one of the data platform tools that has:

• An integrated UI and control plane for seeing what’s going on, unifying all your tools into a single webpage.
• Lets you see your data assets in a list with extensive metadata: it’s your data catalog showing all tables, BI dashboards, reports, and other data assets.
• Keeps and creates all the metadata when we run data pipelines across all data systems: With end-to-end access, we can also have metadata and data lineage end to end, which helps us understand where the data comes from, and in case of error, where the bad data is.
• Has integrated scheduling, sensors, backfills, and concepts to work with data, built in.
• Supports multi-team isolation through code locations, so different teams can own different parts of the platform without stepping on each other.

As Dagster is open source, it gets promoted from a usual data orchestrator to an open data platform, with the great advantage of transparency: easily patching an error or integration if you need to integrate an obscure system that only your company has, working on the cutting edge with the community, or getting features from them.

[!note] The Venn diagram of Dagster
Obviously you can’t optimize a data platform in all directions. If you look at Dagster as a Venn diagram, it has these three circles: the right abstraction, flexibility, and full automation through programmability.

Control Plane: Center with All Metadata

It integrates multiple different teams such as data engineers, platform and infra teams, with data science and business people who want to run their jobs. Feature-wise, it provides data catalog and contract capabilities, lets you see data assets you’re responsible for and when they last got updated, and shows their downstream and upstream dependencies, all in one real-time observability and monitoring UI.

This is all done through its control plane, which centers all metadata and unifies different data tools along the lifecycle into one platform, something usually only closed-source data platforms achieve. The control plane serves everything in a single view, showing how all processes are working. It’s the operational dashboard for your company.

With data orchestration as the heart of data work, with metadata for any process and access to all source systems we’re pulling from or intermediate systems, it’s in the perfect place to serve as the central metadata store. Think of INFORMATION_SCHEMA, but for overall data work, not only one single database. Only the orchestrator can understand the system and its status this deeply.

Why Open Data Platform: a System that Unifies Open Source

The open in open data platform is interesting, as it’s really hard to build a unifying layer across different stateful data systems, and it’s worth highlighting that Dagster achieved just that.

If we look at open data stack architecture: unlike cloud data platforms that have the same goal, such as Fabric, Snowflake, Databricks, etc., Dagster builds on open standards and is itself an open standard. It’s like a protocol in which we declaratively define our data assets (e.g. Software-Defined Assets, environments, resources) that then get automatically executed with composable computes we define in resources, all interchangeable. Even closed-source engines such as a Databricks Spark cluster work really well.

The hardest part is integrated data governance, lineage, access rights, and compute. An orchestration platform like Dagster doesn’t give you everything, but you get most of what you need, in an open and composable way.

Composable Data Stacks Possible

What this architecture allows is what Wes McKinney calls “Composable Data Stacks” in Monday Morning Data Chat, essentially rebundling the data platform.

Composable data stacks depend on compute engines. With this architecture, plus Dagster resources or external code (Dagster Pipes), we can easily pick and choose what is best suited for the task at hand, not only for different jobs but also depending on different environments. Although there will never be one singular tool for everything, it’s necessary that we have a layer of integration, and there’s no better place than the orchestration layer that separates execution and technical logic from business and already deals with multiple data environments.

Pete Hunt, the CEO of Dagster, said recently:

Our goal is to make AI as lightweight, accessible and cheap as we can to drive enterprise value, while enabling our customers with the infrastructure they need underneath - the orchestration and data platform layer where reliability, support, and recovery actually matter.

Pete is reaffirming that the data platform layer is the priority, so AI can build on top of a great foundation. In the eyes of a data engineer, it’s a dream come true to have an open platform that is declarative to simplify the overall architecture, but also uses AI for the glue code, based on best practices enforced through Dagster as the open data platform.

The Right Abstraction Layer for an Open Data Platform

Charlie Munger’s Almanack distilled decades of investing wisdom into timeless principles and mental models that compound over time. Here I tried the same for Dagster.

The principles touched on won’t be obsolete next year. Data-aware orchestration, declarative assets over imperative DAGs, separation of business logic from infrastructure, and composable stacks with a single control plane are all mental models for building data platforms that hold up whether you’re running DuckDB on a laptop or Spark across three cloud providers.

Eight years after discovering Dagster on a podcast during my time in Copenhagen, I’m still reaching for it whenever a system gets complex enough to need real orchestration. With extensive built-in data quality checks, best practices like unit-testing, local development to prod, separation of business and technical logic, code locations, Domain Specific Languages (DSLs) for non-technical people, pipes and components to run something in Rust or Go, and many more, the Dagster data platform gives you huge leverage building from strong foundations, with the flexibility to change along the way.

It’s the abstraction layer for data engineering to solve hard business problems, an open data platform with opinionated design decisions that compound the longer you build on them.

In the next piece, I’ll get into what it actually takes to operate this — architecture, deployment, and governance — as a follow-on to these principles.

Next Steps

Find Dagster’s official skills for the latest and most updated way of working with Dagster, to feed to your AI agent. Or read the blog post with more information. If using Airflow, migrate from Airflow, or use Airlift for an integration for legacy and critical DAGs still in Airflow.

Find awesome-dagster, and check out further readings of mine at Data Integration as Code: Configuring Airbyte and dbt with Python (Dagster) or Data Orchestration Trends: The Shift From Data Pipelines to Data Products.

Want to use all of this stress-free without the deployment burden? Use dagster+. Great tradeoff between cloud and OSS, still having the OSS Dagster foundation, but profiting from extra features (GitHub integration, cloning, etc.) and not needing to set up a DevOps pipeline or fiddle with Kubernetes.

Full article published at Dagster.io - written as part of my services

Chris has seen the data stack evolve over the years. He thinks AI will soon handle the majority of data engineering work, provided with the right tooling and access to CLIs and APIs. He also thinks LLMs might not speak SQL, but a format that represents data transformations. With so much shifting and changing currently in the AI space, new models, new workflows weekly, Chris’s perspective helps you navigate without overreacting, based on a long experience in the domain.

The article is structured in four parts: (1) correctness when working with financial data, (2) the Ralph Loop and why AI might be better off speaking something other than SQL, (3) vulnerabilities and the case for “Okta for Agents,” and (4) the future of AI — including why “data engineer” as a distinct role might not survive.

Introducing the Guest: #2 Chris Riccomini

Chris Riccomini is a Software Engineer, Author, Investor, and Advisor. Previously at WePay, LinkedIn, PayPal, and author of The Missing README: A Guide for the New Software Engineer and co-author of 2nd version of the iconic Designing Data-Intensive Applications book.

Chris has been working in open source throughout his career. He is the author of Apache Samza, a distributed stream processing framework. His current project is SlateDB, an embedded key-value store built on object storage. He is also on the Apache Airflow’s PMC.

Correctness of Data in the Financial Sector: How Does This Work with AI?

Chris had worked at financial companies where data correctness is essential. My first question was “How do you see using AI in data when financial services, or most other places, must be correct? How do you mitigate the small errors AI still makes in such a situation?” His response:

It really depends on where in the stack AI is being deployed.

Use Cases with Different Risk Profiles

Risk, fraud and compliance. The bar is model explainability, you need to know why the model made the decision it did:

If AI is involved in decisioning around risk and fraud, compliance and “model explainability” comes into play (why the model made the decision it did). This is one of the reasons we really liked random forest models at WePay: you could explain the actual rules that the model had derived and used in order to make a decision.

The data engineering context, compared to a traditional data modeling situation, is interesting:

If AI is being used in a data engineering context, it seems to me more like a traditional data modeling situation. You should be able to define invariants that must always be true for your data. For example, the ledger should always sum up. This is how we managed our data pipelines. If AI is defining data integration pipelines and moving data, the invariants should still hold. Traditional data verification tools will continue to play a role there.

For data analytics, this is where most of the fear lives:

There is a fear that AI will hallucinate and cause a bad decision to be made. I think this is a reasonable fear, but it’s also a problem we had before AI. Data in any organization is messy. Semantics aren’t always clear, contracts get broken, and so on.* *Every company I’ve worked for has had this problem. It’s not uncommon to find an incorrect query that’s been rolled up into a weekly ops review with the CEO, for example. This was true before AI.

So the question, is whether AI makes this worse or better. Chris own view has shifted recently:

If you’d asked me two years ago, I would have said it was definitely going to get worse. Now, I think it might actually get better, especially if we pair AI with a human. The latest LLMs have gotten really good at spotting bugs, inconsistencies, and so on. My personal experience is that I’m both more productive and more accurate with an AI.

I am having a similar experience: for working data engineering projects, if I use it for a not-too-distant future, meaning if the scope is clear and in a framework or rigid structure, it can implement a great solution since last December 2025, when the models got better. With it, it can go a long way, but still, it can’t work autonomously, or do a full project from scratch. It still needs a lot of hand-holding, as it does not understand the business.

So, balancing quantity with quality and keeping up with reviews at the speed of generation is also a challenge, especially since the model usually generates many lines of code. But for my writing process, where my personal voice plays a bigger role, I find that AI can’t help me too much yet in the actual writing process - but on the surrounding tasks (research, brainstorming, though also limited for new topics that are not based on existing ideas).

LLM Should Speak Substrait, not SQL

Chris said recently that: “Similar to my belief that LLM should speak substrait, not SQL”. I asked him to explain this quote and he said:

This is more of an intuition than something I’ve demonstrated to be true. But if you look at the way we use SQL, it’s actually used in two different ways: by humans and by machines. I think both can benefit from Substrait (or some equivalent).

Chris continues to explain that “Substrait is a format that represents data transformations. It has many operations that SQL has, but unlike SQL, which is purely logical, Substrait lets you define physical operations as well. In SQL, you say JOIN, but in Substrait you can say how to join: merge join or hash join? For those with a compilers background, Substrait can express both abstract and concrete syntax trees–intermediate representations (IRs).”

This is valuable for LLMs for two reasons:

1. You should be able to express SQL with fewer tokens (provided the serialization format for the logical operations is more efficient than english). This should make LLMs slightly cheaper to use, but more importantly it should keep them from hallucinating quite as much. (Granted hallucinations are less of a problem than they used to be).

2. More importantly, LLMs are pretty smart. They should be able to do query optimization really well. And Substrait gives them that ability–they can express physical operators (e.g. merge vs. hash), not just logical ones. This should allow them to do query optimization on the client side, and pass a physical query plan directly to the DB for execution (provided they have access to the requisite table statistics).

Substrait, as an emerging standard that provides cross-language serialization for relational algebra, is very interesting and something I want to check out, especially the expressiveness compared to SQL.

[!note] Downside of Substrait: LLMs are less familiar
Of course, there is a ton of SQL on the internet, so it’s not clear that LLMs will be as amenable to working with lesser known formats like Substrait. I think it’s worth experimenting with, though.

Making AI Output More Reliable

What I learned is that the longer something is in the future, the more vague or incorrect or hallucinated the outcome can be. So the more context and code you can provide, the more accurate the result. Which is pretty much in line with Substrait.

But how do we work with the LLMs, what’s the best approach, using god mode in OpenClaw or --dangerously-skip-permissions in Claude Code with no limits where it can go indefinitely with not much more context? I asked Chris if that’s also what he observed, and if he uses plan mode and a declarative approach or pipelines, as it helps for context and collaborating with the AI on a shared output, usually Markdown.

I was having coffee with a friend of mine, lamenting about this very problem a month or two ago. I was trying to get Codex to do something complex and it just kept falling on its face. My friend told me that you have to live in plan mode all the time. You can’t just ask it to plan the work, then flip to “Implement this plan.” You need to have the LLM iterate on the plan for many iterations. Probe its plan, ask it for details, ask it to expand sections, and so on. You need to get to the point where you feel like there’s no possible way the LLM can’t implement the plan incorrectly.

[!info] Spec-driven AI work

On the note of “needing to have the LLM iterate on the plan for many iterations”, Mark Freeman suggested in previous interview the spec-driven development (SDD) approach with the open-source GitHub Spec Kit, check it out or read the previous interview for more context.

The Ralph Loop: And Managing Context

After having a plan at hand, the next step is to keep the LLM’s working memory lean:

Once you have a good plan, you need to manage context. In some cases, you will need to take your plan and start with a fresh context in the LLM. In other cases, you’ll need to clear the context periodically throughout the work. I use a Ralph Loop for such cases¹.

I had the exact same experience when working with smaller code bases: to refresh context, the insights you gain over the iterations are not as effective if you add them bit by bit, compared to if you refresh memory and start over with all the new key insights provided at the very beginning, steering the model to a more tailored direction earlier on.

But with the Ralph Loop, which refers to understanding AI beyond surface-level applications, you get new insights that you can then add to your initial prompt, that you wouldn’t have gained otherwise, by exploring deeper programmable patterns.

The loop is an iterative, autonomous AI development technique where a bash loop (or plugin) repeatedly prompts an AI agent with the same goal, forcing it to persistently iterate until tasks pass external tests. It forces the AI to work, fail, and fix errors until success, rather than relying on the AI to decide it is finished.

On top of that, Chris says “You also need to impose a lot of quality gates. As with plan mode, you need to overdo it. ‘Quality’ is a bit of a squishy term’”, and he breaks it into three steps:

1. Define what quality is for your use case.
2. Measure the quality.
3. Enforce thresholds (gates) that your LLM must adhere to.

[!example] Example by Chris with the three steps for assessing quality

For example, part of your definition of quality might be test coverage. So that’s step 1. Then you set up a coverage tool for your codebase–step 2. Then, you put the phrase, “You always run tests before commit and keep test coverage above 90%,” in your CLAUDE.md. Finally, you install a git commit hook that enforces this rule.

This is a very rudimentary example, but you get the idea. There are a ton of different things you can measure and monitor for your work. I enumerate many in the post Code Quality Gates for Vibe-Coded Projects.

This essentially means we as the Prompt Engineers need to make sure that the workflow is correct, that we understand what we need to do, and accordingly adapt the workflow to get better code quality.

What about Functional Data Engineering, and Executing Deterministically?

In related terms, just as AI might hallucinate, it also might generate different outcomes with the same questions and same context. It’s non-deterministic. But data engineering works especially well if it’s done reproducibly, so we can backfill our data pipelines reliably and trust they will fill the same way.

This also ties into functional data engineering, running jobs with reproducibility and idempotent. I asked Chris what he thinks about this dilemma.

I’m not as worried about this as I used to be. A lot of tooling has popped up or evolved to help address this. Durable execution frameworks try to address some of this by papering over the non-determinism to keep replays deterministic by skipping the previously-successful parts of the flow. Ditto for traditional workflow orchestration systems like Airflow, Prefect, and Dagster. (Disclaimer: I have some Prefect shares.)

Moving to Incremental-loads for Better Determinism?

What I found interesting was Chris’s next suggestion: moving to smaller data sizes, and therefore to loading incrementally for a more reproducible outcome.

We can also move from full batch data processing to incremental batch data processing to help eschew some non-determinism.

A concrete example, splitting load by day:

Imagine, you have a bulk load job that always loads a full table from PostgreSQL into Snowflake, and that job does some LLM-based processing. Every time you re-run it, you’re going to get non-deterministic output. But if you convert it to an incremental job that runs daily and always loads the previous day’s data, then a re-run will only introduce non-determinism into the last day’s load. And presumably you’re re-running that day because something went wrong. In such a case, non-determinism is likely acceptable.

This is great thinking and shows it’s all about the use case and the risk appetite. If you have a lot less back reloads daily, compared to a full load, the accepted risk of one day might be acceptable, if you get great insights from the LLM, or something you’d need to do manually and then the alternative would be you either don’t do it at all, or very late when the insight is “less” valuable.

Side note, the engineering implementation of incremental loads might be much higher than a full load, as you need to add clear state management, checking what has run, and manage that state yourself, versus just running all. But this point almost certainly comes up in any case, whether you use AI or not, so we can factor out that fact in this scenario.

How to Prevent Vulnerabilities, and Work Securely with AI Agents?

Another hot topic with agents is security concerns around vulnerabilities. I asked Chris how he sees that domain in combination with generative AI, and also if we need “Okta for Agents”, as Maxime Beauchemin called it.

His view splits cleanly in two:

On the one hand, it’s a nightmare to manage these agents in the enterprise. On the other hand, they’re phenomenal at detecting compliance violations: leaked credentials, leaked PII, and so on.

He’d been thinking about an Okta-for-agents independently:

It’s funny you mention Maxime’s “Okta for agents” comment. I didn’t see it, but I’ve been saying the exact same thing. It seems patently obvious to me. What’s unclear is whether Okta is Okta for agents, or whether another company (or companies) will take its place. Innovator’s dilemma and all. Okta’s certainly give it a good try–their homepage is covered in it now.

Skills, Marketplaces and MCPs

He continues and says that it’s the wild west right now. You can load skills and even arbitrary skills from a marketplace and load any kind of text files without knowing if there’s a vulnerability.

There are examples where hidden code injection is done in a repo:

Chris continues with not having enough guardrails:

But yes, we absolutely need lineage, auditability, RBAC, ABAC, and so on. It’s the wild west right now (as far as I know, anyway). This is one of the reasons I was so outspoken about MCP when it first came out. I was very disappointed in their (lack of) security model. It’s the most important part, and it was completely lacking. It was rather shocking to me given Anthropic’s focus on the enterprise. More recently, they’ve added better support, though, so credit where credit is due.

Future with AI Agents

When asked about the future of AI, especially when we talk about data engineering, we discussed three interesting topics on what agents are doing well today, the role of data engineering itself and what programming language to use.

What Agents Already Do Well Today

I asked if we get self-healing data pipelines, so we do not need to get up at night, meaning AI does not only detect errors, but also analyses, debugs, pushes a commit to the repo and re-runs the pipeline autonomously?

I’ll be frank: I think AI will do the majority of the data engineering work in the future. I think we’re already at a point where it can; the tooling and practices just haven’t yet adapted.

This is an interesting point regarding tooling (and practices) not being adapted yet. Jeff Dean, Chief Scientist at Google DeepMind, made the point that Amdahl’s Law still applies, and that we need to re-engineer our tools as they were designed for human speed. If AI agents can run 50x faster, but the tools don’t, then we do not get an overall improvement.

On the other hand, what agents already do well today:

Agents are already excellent at inspecting failed Github actions, failed workflows, running SQL queries, writing Python–all the things data engineers do. As they get plugged into monitoring systems and begin to auto-remediate, the grunt work of data engineering will get taken over by AI.

And building new pipelines, given the right access:

Agents are also fully capable of adding new data pipelines, provided they have access to infrastructure to do so. If you stand up a fresh Airflow and add connections for all your systems, I’d wager an Agent can set up as many pipelines as you need on it. And if you define the security and compliance policies it should follow, it’ll do so.

Here, in my opinion, it is key that we use declarative and config-driven stacks, like Kubernetes and React are doing, and most modern tooling.

Data Engineering Role Going Away, or Unified?

Continuing on the thread of the future of AI, Chris talks about how shifting left is a movement we had for a while, and where this leaves data engineers as a role:

I’m not sure where that leaves data engineers. The “shift left” movement has been going on for a while. I can imagine a world in which “data engineer” as a distinct role goes away, or is folded back into a more generic data role that includes data engineering, machine learning, data analysis, and so on.

He’s been pushing this for quite some time:

We over-specialized the data space. It might have been necessary, but it isn’t now. So perhaps we’ll see “data” be a single role that encompasses not just data engineering, but analysis and machine learning/AI as well. I think that would be healthy.

Should We Let the AI Agent Choose the Language?

We heard people saying (e.g. Wes McKinney) that they choose programming languages, in this case Go over Python, based on AI, not what the human prefers. He calls it From Human Ergonomics to Agent Ergonomics. That Wes, the creator of Pandas and author of Python for Data Analysis (stay tuned, he will be the next guest for this interview series), chose Go is interesting, and is because its advantages in fast compile-test cycles and painless software distribution are key. Don’t worry, Python will not go away².

Or Ladybird is rewriting part of the browser entirely from scratch in Rust with agents in two weeks. So Chris, do you think that choosing the programming language will depend on the ergonomics of the agents in the future (or now already)?

In a word: yes. I have been pretty enthralled with the software factory concept lately. It’s how I do a lot of my development now. In that world, I just don’t care about the language my software is written in.

What he optimises for instead:

I care more about the characteristics of the output: its performance, stability, and cost to build (i.e. tokens). Languages that lend themselves to faster, cheaper, more stable LLM output are going to win.

These are very interesting thoughts, and I did a project fully vibe coded in Go to experience the cost-as-tokens as well. The codebase kept being small (apart from the tests), and therefore I could go much further with the given tokens compared to other projects where I used the same Claude Plan Pro and ran out.

Go is a language I don’t usually program in. And it is quite astonishing how far you get, but I also noticed a limitation as Lines of Code and size of the project grew, especially when adding new features that would break working features.

[!note] Side note by Chris: No more proofs for a programming language required

I used to think that this would pull us in the direction of languages that have formal methods properties. Proofs, model checkers, and so on. I no longer think that’s the case, though. I think LLMs have gotten good enough at writing code that proofs are no longer required. We can use normal testing strategies. I wrote more about this in The Waymo Rule for AI-Generated Code.

Does AI Take away the Learnings?

Last question I asked Chris — the danger of not learning new things, and getting overwhelmed with constant stimulation, and even addicted? In a world where we only prompt, where we don’t experience hitting a wall and then figuring it out, does that prevent us from learning new things? Are we just cruising on auto-pilot?

Chris mentions that it depends on how we use it and brings an example:

One could argue a calculator makes us learn less math; indeed, I keep an eye on that with my middle school-aged kids. But it’s also a tool that lets us do far more complex math without worrying about carrying the one or shifting the decimal, so to speak.

But you can also learn with AI he argues:

I have had instances where I learn a ton from AI. A concrete example: SlateDB’s language bindings. I built them all from scratch (or rather, AI generated them all from scratch). When I started, I knew nothing about bindings. As I worked with AI to steer it and iterate on the code, I learned about cbindgen, UniFFI, foreign function interfaces (FFIs), and so on. It’s a phenomenal tool for picking up something from scratch. I can ask it questions, learn from it, and so on.

Again, did he actually learn as much (from scratch, with AI) as he would have building it himself?

Almost certainly not, I think I would have learned a lot more [without AI]. But I also wouldn’t have done the work. Writing four bindings (Node, Java, Python, and Go) from scratch is just too much work. I don’t have the time for it. Especially since I have never written a line of Go, and I know next to nothing about the Node ecosystem. So in the real world, I think I came out ahead.*

Do We Learn Fewer Things?

Let’s finish with a question: Are we learning fewer or just different things? Something I’ve wrestled with for a while. Chris’s answer is:

Perhaps the things we are no longer learning don’t really matter anymore. Going back to the calculator example, I couldn’t really tell you in detail how a calculator physically works. If you took it apart and showed me its circuitry, I’d be unable to tell you anything about it, really. Does that matter? I’m not so sure.

I think we all are in this experience together, and nobody can really predict the future. I experienced both sides: when I rely too much on the assistant, I get more lazy and do the deep thinking less. While I course-corrected, and only used it for dedicated tasks, I noticed that abilities were improving again, or better, my feel and gut feeling got better again, and I had more confidence in the task at hand. But also, as Chris said, if I know it’s going to be a hard task, I can do much more because I deliberately use AI for certain tasks to actually finish the task. So the future will tell.

Next Interview

I hope you enjoyed this interview with Chris. Huge thanks to Chris for taking the time to speak with me and for sharing his experience with all of us. Follow him on LinkedIn, X/Twitter or on Bluesky, read his two amazing books. Follow his amazing newsletter, the new one at Posts on engineering, venture capital, AI, and more. | rng.md, but also his old one Materialized View has a wealth of insights.

There are three more interviews already lined up with great guests, one of them is Wes McKinney as mentioned, so please share feedback, questions you might want to ask or just your experience on how to work with AI in the data space. We’re all in this together, figuring it all out. The more we can learn from each other, what’s important, and maybe also what’s not, the better.

So stay tuned for the next interview.

Full article published at MotherDuck.com - written as part of my services

The core question is: “When should you store data internally in your warehouse versus externally in object storage?”. Hot data queried frequently goes inside. Cold archival data stays external, where it’s cheaper but slower. Interestingly, Databricks and BigQuery recently added external table features, but why? Not because they’re trendy, but because the economics still work.

This article offers an inside look at external tables, their 25-year history, how they evolved from CSV parsers to ACID lakehouse tables, and whether you need to know about them today.

What Are External Tables?

So what are external tables, and why have we been using them for so long? Why don’t we just use the internal storage of a database?

In Oracle, where I first used them in 2008, they allowed you — and still do — to access data in external tables. External tables are defined as tables that do not reside in the database, and can be in any format for which an access driver¹ is provided. All of this is provided via DDL (Data Definition Language) of the database, describing an external table with all its columns, data types, etc., exposing the data as if it were residing in a regular database table.

The external data can be queried in parallel and queried directly using SQL. Essentially, it’s read-only access to data stored outside of our database, making it available in a tabular, easy-to-work-with format to interact with existing tooling and language. In 2008, this was through procedural language such as PL-SQL in Oracle or T-SQL on MSSQL.

Today, external tables have evolved. The biggest change is that they can read more formats including semi-structured data such as Parquet, JSON, Avro, and ORC. While CSV was readable in 2008, the difference today is the columnar formats and nested formats that enable faster analytics. These are available for downstream processes and dashboards, but mostly accessed through SQL queries in one form or another.

A modern definition by BigLake, an evolution of BigQuery toward a multi-cloud lakehouse that tries to solve key customer requirements around the unification of data lake and enterprise data warehousing workloads, introducing external tables in 2015 as part of it²:

External tables are stored outside of BigQuery storage and refer to data that’s stored outside of BigQuery. [..] Google Non-BigLake external tables let you query structured data in external data stores. To query a non-BigLake external table, you must have permissions to both the external table and the external data source.

Snowflake defines them as:

[…] When queried, an external table reads data from a set of one or more files in a specified external stage, and then outputs the data in a single VARIANT column. Additional columns can be defined, with each column definition consisting of a name, data type, and optionally whether the column requires a value (NOT NULL) or has any referential integrity constraints.

External tables were added in 2021, and Snowflake described their benefits as follows:

External Tables Address Key Data Lake Challenges:

1. To augment an existing data lake. [..] augment their existing data lake, rather than replace it. The External Tables feature enables that use case. Customers can use external tables to query the data in their data lake without ingesting it into Snowflake. (side note: MVs³)
2. Ad-hoc analytics. Customers often use external tables to run ad-hoc queries directly on raw data before ingesting the data into Snowflake. Ad-hoc queries help them evaluate data sets and determine further actions.

Just a Pointer (Symlink)?

A simple analogy is a symlink in Linux, where you point from your current directory to another directory without moving data. You just add a pointer. If you read that file from that symlink, all it does is read it from the location the symlink points to.

An external table is the same, just a pointer to external data, bringing that data into the current data warehouse or cloud solution, hence the word external. You define the source format such as XML, CSV, etc., and define their structure, and then you can query that at any time. It’s similar to a SQL View in that sense, but pointing to non-internal data.

Running DROP TABLE and deleting an external table is metadata-based only. No data is removed, only the table definition from the internal data catalog. The same is true with a symlink. Almost any relational database today has support for it, even if it’s not called an external table. Everyone occasionally needs to read data outside of its warehouse or database.

Recap in the History of External Tables

Looking back at the history and evolution of external tables, we can quickly see that there’s a long history and they’ve been a recurring pattern across every generation of database technology since the early 2000s, and arguably longer if you count IBM’s federated database concepts from the late 1990s.

The Origin Story: ISO in 2001

The history starts with ISO/IEC 9075-9, published in 2001. Part 9 of the SQL standard defined foreign-data wrappers and datalink types for managing external data from within SQL. The work was completed in late 2000 and published alongside SQL:1999, with full integration in SQL:2003 (it was later updated in 2023).

It was the initial definition and extensions to database language SQL to support management of external data through the use of foreign-data wrappers and datalink types.

My first encounter was with Oracle external tables, but according to Wikipedia there were earlier implementations, such as Microsoft Access linked tables (~1992). Microsoft Access linked tables (~1992) were the earliest consumer-facing implementation where users could link dBASE, Paradox, text files, and ODBC sources as if they were Access tables. ODBC 1.0 (1992) itself established the first standard for heterogeneous data access across databases, though it didn’t create table abstractions.

Further, IBM’s DB2 DataJoiner (~1995) was more ambitious with a middleware product enabling SQL queries across Oracle, Sybase, SQL Server, Informix, Teradata, and even VSAM files through a unified interface. With SQL Server 7.0’s Linked Servers (1998) we got federated querying to Microsoft’s ecosystem via OLE DB, supporting cross-database joins with four-part naming conventions.

Most of these implementations shared a common limitation that Oracle (9i Release 1 - 9.0.1 in 2001) solved: they focused on querying other databases or required middleware. Oracle’s abstraction treated local flat files as first-class read-only table objects using the familiar CREATE TABLE ... ORGANIZATION EXTERNAL DDL syntax, providing a simple way to define external files as part of normal table creation and allowing ORACLE_LOADER access to query flat files (CSV, fixed-width, delimited) through DBAs.

It was an early way of separating declaration from compute (the Oracle loaders).

Why External Tables? What Are Their Benefits?

But why use external tables? What makes them so useful that they persisted? Why have they survived so long, and why are they getting added to Databricks and other major platforms?

For that, we need to look at external tables’ benefits. The first reason is that external tables can simplify data access to avoid developing ETL pipelines, moving data out of the source, and re-ingesting it in our data warehouse. They make external data accessible easily, defined in a tabular form by a database schema with column types. Typical cloud data warehouses like Snowflake and Azure use them to link existing data from object storage easily without moving data. This makes the object storage files accessible for almost any downstream tool or query language in a simple and cost-effective way.

Other ways of using them are to store some data on cheaper storage (e.g., object storage over data warehouse storage) and only link them in. It’s slower to fetch, but more affordable to keep. If you have large data sets, cost savings can be immense as this article shows, bringing down Snowflake internal storage cost from ~$23/TB/month to S3 infrequent access with ~$12.50/TB or S3 Glacier Deep Archive with only ~$1/TB.

Another handy side effect as the consumer of external table data is that the data is always up to date, because no refresh or update is needed. It goes without saying that this has its own downsides and can be a problem for the owner of the data if it’s used in production and the ETL process reads large amounts of data through external tables. This will affect upstream apps running or owning this data.

That’s why many use external tables in combination with materialized views (MVs) to truncate and recreate a daily snapshot (or similar) during off-peak (mostly nights) of this data, avoiding affecting production data and even optimizing query performance with added indices for downstream queries.

When Internal and When External Data? What’s the Limit of External?

The tradeoffs come down to how often the data is queried, e.g. the hot versus cold question.

The tradeoffs and considerations you should make when wanting to use them come down to the decision of how often the data is queried. The table below shows it in more detail:

In the era of data lake and lakehouse architectures, this is an important consideration. VSCO says: “disk space was growing more quickly than our compute needs,” which is what triggered the adoption of external tables.

If you look at your use case, if you need to do analytics across various sources with joins and augmentation of your data at an enterprise, you probably want to focus on loading data into your database or data warehouse, an architectural pattern that has survived more than 30 years. But if you have data that is external and small but you want to join it with existing data, or you always need fresh data and can live with a slower response time (maybe because it runs during the night), you might use external tables.

In any case, external tables are a good approach to keep in mind and a valuable toolkit to have.

They Work Well with Existing Tech and Common Patterns

Obviously, today’s external tables are not the same as the earliest ones in Microsoft Access, but the principle of accessing data outside your system is still the same. Nowadays we have more support, new formats besides CSV and JSON. We can do Parquet or open table formats.

As mentioned, they work well with related long-lasting data warehouse patterns and applications such as materialized views and stored procedures. The recurring pattern is to access external data with your data management system, similar to the pattern of materialized views that refresh complex SQL statements and make them fast, and stored procedures that run glue code within your database.

Moreover, there are temporary tables that are similar but only available during a transaction or session. They all work in the same Lindy effect, e.g., Databricks just announced Temporary table support recently on December 9th, 2025, or Databricks SQL Stored Procedure a little earlier, August 14th, 2025, for reusing existing SQL statements.

Again and again, everything that is old will be new again. Exactly what the Lindy Effect is all about. We can clearly say that the Lindy effect over the last 33 years applies here. The longer something is in place, the more likely it is to be around for at least that long.

[!info] External vs. Temporary Table

In contrast: temp table = session-scoped, writable, fast, invisible to others, auto-dropped. External table = persistent metadata, read-only, infinite size, visible to all, optimized for cost.

A common chain in practice is going from: external table → temp/transient table → permanent managed table.

How a Classical External Table Works

To understand how traditional external tables work, let’s first look at Oracle, which has built an extensive syntax around them and where they still work this way today.

First, we can create a place for external data called DIRECTORIES, which is simply a pointer or alias to a file system location where external files already exist:

1
2

CREATE OR REPLACE DIRECTORY admin_dat_dir
    AS '/flatfiles/data';

This directory can point to local file systems, NFS mounts, or even cloud object storage today (with the ORACLE_BIGDATA driver for S3, OCI, Azure). The DIRECTORIES don’t require moving data, though you could prepare those files via ETL pipelines or third-party tools, or they can be generated directly by applications.

We can now create an external table based on this directory, e.g., log files, bad data that we store externally, JSON files, and make data accessible inside the INFORMATION_SCHEMA and with plain SQL, as if it were internal.

Creating an external table:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

CREATE TABLE admin_ext_employees
                   (employee_id       NUMBER(4), 
                    first_name        VARCHAR2(20),
                    last_name         VARCHAR2(25), 
                    job_id            VARCHAR2(10),
                    manager_id        NUMBER(4),
                    hire_date         DATE,
                    salary            NUMBER(8,2),
                    commission_pct    NUMBER(2,2),
                    department_id     NUMBER(4),
                    email             VARCHAR2(25) 
                   ) 
     ORGANIZATION EXTERNAL 
     ( 
       TYPE ORACLE_LOADER 
       DEFAULT DIRECTORY admin_dat_dir  --notice this dir with above
       ACCESS PARAMETERS 
       ( 
         records delimited by newline 
         badfile admin_bad_dir:'empxt%a_%p.bad' 
         logfile admin_log_dir:'empxt%a_%p.log' 
         fields terminated by ',' 
         missing field values are null 
         ( employee_id, first_name, last_name, job_id, manager_id, 
           hire_date char date_format date mask "dd-mon-yyyy", 
           salary, commission_pct, department_id, email 
         ) 
       ) 
       LOCATION ('empxt1.dat', 'empxt2.dat') 
     ) 
     PARALLEL 
     REJECT LIMIT UNLIMITED; 

The first and most important choice is TYPE, which determines the access driver and what kind of files you can read: ORACLE_LOADER for plain text files like CSV or logs (read-only), ORACLE_DATAPUMP for Oracle binary dump files, ORACLE_BIGDATA for cloud object stores like S3 or OCI in formats like Parquet or Avro, and ORACLE_HIVE for Hadoop/Hive data. The DEFAULT DIRECTORY points to a server-side path alias, and LOCATION names the actual file(s), with wildcard support (*.dat) so you can load a whole batch at once.

The ACCESS PARAMETERS block is where you control parsing: row and field delimiters, null handling, custom date format masks, and where to write bad rows (badfile) and parse logs (logfile). On top of that, PARALLEL lets Oracle split file reading across multiple processes for large files, and REJECT LIMIT controls fault tolerance. Set it to UNLIMITED to skip bad rows silently, or 0 to fail immediately on the first error.

You see lots of built-in features that we can use compared to building a full-fledged data pipeline. Instead of exporting and importing CSVs from the source databases or developing a complex CDC pipeline that traditionally looked something like: source OLTP --> CSVs --> IDW (reports on yesterday) -> ingest into DWH for long-term analytics, we can just define a table based on external data and access it as part of our pipeline.

[!tip] The INFORMATION_SCHEMA analogy

You are probably familiar with the INFORMATION_SCHEMA of a database. It’s the internal data catalog that most databases provide and it contains a list of all tables and all metadata such as columns, data types, etc. The neat thing is that external tables will show up as internal tables once defined.

What’s the Modern Version of External Tables Today?

To preface: the previous Oracle example shows the CREATE EXTERNAL TABLE syntax, and a first-class DDL object in the data catalog. What follows in this chapter is the next evolution, where external tables are not necessarily created with DDL, but in another way, achieving the same outcome of querying data in place without loading it. Let’s see what these are.

Integrated into Warehouses

Most modern warehouses - Snowflake, Redshift Spectrum, BigQuery, Athena, Synapse - come with a simplified version of CREATE EXTERNAL TABLE. Compared to the Oracle example, the schema is usually inferred from the file format (especially Parquet), S3 or another object store is the default backing location, and the parsing ceremony disappears. The pseudo-code looks roughly like this across engines:

1
2
3
4
5
6

-- Pseudo-code: modern external table over Parquet on S3
CREATE EXTERNAL TABLE sales
WITH (
  LOCATION = 's3://my-bucket/sales/',
  FORMAT = 'PARQUET'
);

Object storage like S3, GCS, and Azure Blob has become the first-class citizen for external data. From here, the ecosystem layers on: dbt wraps this in YAML, DuckDB skips the DDL entirely in favor of schema-on-read, and open table formats add transactional guarantees on top.

External Tables with dbt?

On top of this base SQL form, dbt adds a YAML layer and can be used with its own package called dbt-external-tables. It’s one of the most-used dbt packages, though it seems less actively maintained now.

The external table is defined via YAML, and there are lots of options to set, with the most important being external and its location, but also defining columns in different ways such as inference or the meta tag:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

version: 2

sources:
  - name: snowplow
    tables:
      - name: event
        description: >
            This source table is actually a set of files in external storage.
            The dbt-external-tables package provides handy macros for getting
            those files queryable, just in time for modeling.
                            
        external:
          location:         # required: S3 file path, GCS file path, Snowflake stage, Synapse data source
          ...               # database-specific properties of external table
          partitions:       # optional
            - name: collector_date
              data_type: date
              ...           # database-specific properties

        # Specify ALL column names + datatypes.
        # Column order must match for CSVs, column names must match for other formats.
        # Some databases support schema inference.

        columns:
          - name: app_id
            data_type: varchar(255)
            description: "Application ID"
          - name: platform
            data_type: varchar(255)
            description: "Platform"
          ...

        # Use `meta` to pass custom column properties (e.g. alias, expression)
        columns:
          - name: raw_timestamp
            data_type: timestamp
            config:
              meta:
                alias: event_timestamp       # rename the column in the external table
                expression: TO_TIMESTAMP(...) # custom SQL expression instead of default value extraction

This is a nice improvement over the ODBC GUI interface. It’s not exactly an apples-to-apples comparison as dbt itself is not a database, but with its supported destinations such as Redshift (Spectrum), Snowflake, BigQuery, Spark, Synapse, and Azure SQL, you see that it will persist in these destinations, mostly data warehouses.

DuckDB with dbt

If you use dbt, you can also use DuckDB with dbt via dbt-duckdb, which is more up-to-date. But DuckDB is not an external table, right?

Yes, DuckDB doesn’t have CREATE EXTERNAL TABLE syntax yet, mostly because it is an in-memory database, but you can achieve the same functionality through other means. DuckDB can not only be used as a database but also as a zero-copy SQL connector (see all categories at 5 Key Categories). We can just point it to an external source, as shown above with dbt. The difference is that DuckDB is both a database and a compute engine, making ad-hoc reads possible directly without a DDL definition, similar to an external table with Oracle loaders. With dbt, we can nicely declare this in dbt configs.

With DuckDB, you can query “external data” extremely fast over HTTPS or locally in formats such as Parquet, CSV, and many more, so the need for formal external tables is reduced since DuckDB does schema on read.

If you want to define the database schema ahead of time, we’d use external tables to do that and effectively have schema on write (though we don’t write, just define the DDL table structure and data types), which is more of the classical ETL approach.

Here’s an example with external_location to read external data with dbt:

1
2
3
4
5
6

sources:
  - name: external_source
    config:
      external_location: "s3://my-bucket/my-sources/{name}.parquet"
    tables:
      - name: source1

Read more at Fully Local Data Transformation with dbt and DuckDB.

Other options are with database views that are supported in DuckDB with CREATE VIEW over read_parquet(). You can ship a .duckdb file to clients with pre-defined views over S3 data, so clients don’t need to know about the underlying data, Hive partitioning, or even glob patterns — very similar to what a formal CREATE EXTERNAL TABLE would do.

1
2

CREATE VIEW events AS
  SELECT * FROM read_parquet('s3://lake/events/*.parquet', hive_partitioning=true);

Or similarly use ATTACH to directly point to Postgres, MySQL, SQLite, S3, and others:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

-- Postgres (binary wire protocol, predicate + projection pushdown, read+write)
INSTALL postgres; LOAD postgres;
ATTACH 'dbname=postgres user=postgres host=127.0.0.1' AS pg (TYPE postgres);
ATTACH 'postgresql://user@host/db' AS pg (TYPE postgres, READ_ONLY);

-- MySQL (via MariaDB Connector/C; Postgres-style keyvalue string even for MySQL — easy trap)
INSTALL mysql; LOAD mysql;
ATTACH 'host=localhost user=root port=0 database=mysql' AS mdb (TYPE mysql);

-- SQLite (file opens directly; multi-reader single-writer by SQLite file locks)
INSTALL sqlite; LOAD sqlite;
ATTACH 'sakila.db' (TYPE sqlite);

-- Generic remote DuckDB file
ATTACH 's3://duckdb-blobs/databases/stations.duckdb' AS stations_db;

Open Table Formats and Lakehouse Architecture

That begs the question of whether Open Table Formats are the next evolution and modern way of external tables. These table formats allow almost any SQL compute engine to use them as external tables, and read, compute, and aggregate as a database would.

If we look at what table formats consist of, they’re built on object storage, with a file format like Parquet, and then we have a manifest file that contains a list of files that unifies multiple single files into a “single” table, looking from the outside.

So again, the manifest file is our pointer or fancier symlink, but it lives next to the data, unlike external tables. There’s much more going on in table formats, but if we have a data lake with open table format tables, we can see how we define tables in DDL and the pointers are to different files (Parquet, ORC, Avro), in most cases Parquet.

More broadly, we can say external tables decouple storage from compute. Open table formats decouple the table itself (schema, history, transactions, statistics) from any single engine.

Lakehouse and Connecting to DuckLake

One step further is obviously a lakehouse architecture, with the shift from format-agnostic file reading to governed, transactional, multi-engine open table formats.

If you extend the external table idea to a lakehouse architecture, these external tables with open table formats provide essentially what databases provide with ACID guarantees, time travel, schema evolution, partition evolution, and fine-grained access control, but for files.

But with the difference that data stays in open Parquet file format on customer-owned cloud storage. The external table, once a humble workaround for avoiding data loads, has become the architectural foundation of the data lakehouse if you like this analogy.

With DuckLake, we have the next evolution just around the corner, bringing back exactly that missing database, especially to handle all the metadata of such a lakehouse and all its files. This means having durable and consistent database storage for our manifest files.

Open Data Catalog to Complete the Picture: The ODBC Glue

With all these evolutions, we’ve come far. When adding an Open Data Catalog, we are exactly where we started: having an INFORMATION_SCHEMA, a dictionary with all our tables, in this case the open table format tables.

It’s the glue that ODBC provided when connecting a BI tool to the underlying database. Now you’d like to have an open data catalog that, in the best-case scenario, gives you all the tables and ways to connect.

But then again, the syntax of EXTERNAL TABLES still gets added, and ADBC and DuckDB are doing a great job of using external data without needing a data lake and its technology stack altogether. For example, DuckDB has support for ODBC, ADBC and even JDBC. That matters especially for 3rd-party tools: ADBC streams Apache Arrow end-to-end instead of serializing row-by-row, so BI tools and notebooks can pull millions of rows directly from external Parquet tables at speeds that previously required keeping data “hot” in a cloud data warehouse. 🙂

[!note] ADBC, what is that?
ODBC is 30+ years old, and we have a newer, faster version of it, called ADBC. It’s a faster way to connect to other databases with a columnar-oriented API instead of row-by-row serialization, heavily making use of Apache Arrow.

While ADBC is newer, it tries to support the same drivers as ODBC, but faster and easier to install. E.g., it has a handy dbc CLI to install it on almost any programming language, so no more manual and error-prone Windows GUI ODBC downloading of drivers and definitions needed, just one CLI command.

[!tip] Using MotherDuck
If you want a data warehouse that just works, integrates well with DuckDB, and has support for DuckLake, you can always use managed MotherDuck. You can build a classical data warehouse with plain SQL, you can read external data easily with DuckDB or dbt-duckdb, or integrate with DuckLake.

It works great with agents. Check out MotherDuck’s agent-skills for opinionated AI skills for building applications with MotherDuck. And visualize with Dives with one prompt.

Which Is Faster? A Quick Benchmark

To put numbers behind the hot/cold decision, I ran a simple benchmark on the TPC-H SF=1 lineitem table (6M rows, ~150 MB), stored four ways: inside a DuckDB file (internal), as raw Parquet, as an Iceberg table, and as a DuckLake table. Full code: bench2.py and metadata_bench.py.

Dashboard workload (hot path): 3 queries × 10 repeats:

Internal is fastest; external pays a 1.3×–1.7× tax. But for cold/archival queries (one-off, no warmup), all four backends answered in under 150 ms. The speed difference effectively vanishes for data you query once a week.

Storage cost is where external tables shine. Columnar Parquet is ~40% smaller than native DuckDB format. Ten TB of archive data costs roughly ~$125/month on S3 Infrequent Access or ~$10/month on Glacier Deep Archive, versus ~$230/month inside Snowflake on capacity pricing. This is the economic case external tables were invented for, and it still holds.

Metadata workload is where DuckLake stands out. Fifty single-row inserts showed DuckLake creating zero data files (rows inlined in the catalog) versus Iceberg’s 352 files (201 data + 151 metadata). That’s the “small file problem” made concrete: at one write per second, Iceberg creates ~86,400 files per day needing compaction. DuckLake creates zero until you checkpoint. DuckDB Labs’ own benchmarks report up to 926× faster queries on streaming workloads.

So Should You Use External Tables?

So after all this, should you use external tables today? After seeing how sticky they’ve been since Oracle 9i in 2001, how they keep getting re-added to newer tools (Snowflake in 2021, Databricks Unity Catalog, BigLake in 2022), and how their core benefit is. Accessing data where it lives without moving it, via a simple DDL statement, has only grown more valuable as formats have evolved from CSV to Parquet, JSON, Avro, and now open table formats. I’d say yes. But choose wisely based on your data’s temperature: use internal storage for hot data, such as dashboards and frequently used queries.

Use external tables for cold data, archival workloads, and ad-hoc exploration, where that gap vanishes, and storage costs plummet (up to 20× cheaper on Glacier Deep Archive vs. warehouse-managed storage). And if you already use dbt, DuckDB, or a lakehouse stack, the modern versions are right there. Where they’re the wrong choice is the inverse: transactional workloads, queries that need sub-second latency on every run, or data so small that the operational overhead of an external stage outweighs the benefit of not loading it.

The evolution is worth naming explicitly: “read CSVs on disk” → “read Parquet on HDFS” → “read Parquet on S3 via a metastore” → “read Iceberg/Delta tables with ACID on S3” → “the Iceberg table is the warehouse table”. Each step kept the core idea (data stays where it lives, metadata describes it, SQL queries it) and added database semantics back in. With open data catalogs, the warehouse becomes a stateless rental over a bucket you own, and external tables are increasingly managed. DuckLake demonstrates this best: when the catalog has SQL-DB-like guarantees, the distinction between “external” and “internal” dissolves. The metadata benchmark made this concrete by reading a single indexed row rather than walking a manifest tree.

The database semantics are returning with DuckLake, managed Iceberg, and predictive optimization, all of which reintroduce RDBMS-style guarantees to the lake. The cycle from “external table for cheap storage” to “external table as a full ACID database on S3” took 25 years, completing the journey back to database principles while maintaining the separation of storage and compute. You can say the modern external table isn’t external anymore. DuckDB reads them directly, and DuckLake handles the metadata that multifile lakehouse architectures would otherwise drown in. The lesson from history is that whenever someone tries to replace it, the pattern is that reading data in place always beats moving it. And the Lindy Effect suggests that if external tables have lasted 25 years and get re-added, they’ll persist another 25. They’re probably not going anywhere. 🙂

Full article published at MotherDuck.com - written as part of my services