Luckily, Maxime Beauchemin, the creator of Airflow and Superset and the person who defined what “data engineer” meant for a decade (more on him below), joins us to show how he uses agents and what he’s built for working with them. I tried to extract the patterns behind how he actually uses AI in his data work today. This is the fourth interview in ‘How to use AI with DE’.

In this article, we go into four parts: (1) How to balance quality with messy data warehouse work, and how to manage agents with Figma for agents. (2) We elaborate on the future of the context layer and the return to semantics, (3) how Okta for Agents is needed for security, and (4) how the future of agentic workloads can be done in teams, whose yap-to-ship ratio is best, and why Amdahl’s law still counts.

Introducing the Guest: #4 Maxime Beauchemin

Our guest in this interview Max Beauchemin, the creator of Airflow and Superset. He’s known as one of the OGs of defining how data engineering worked back in 2017, and founded Preset, the company behind Superset, and currently serves as its CEO.

He is heavily involved in the AI workflow, which is another reason I wanted to interview him for this series, but he has also been building in the space himself: Agor (Ag: AI agent + Or: orchestration), earlier tooling like claudette-cli¹, and db-agents, an experiment to embed agent context directly inside databases. We’ll get into it all.

Max and I talked about many things, among them how to use AI in data engineering, how security plays a role, how shared, secure, context-rich agent workspaces work within teams, and how he uses AI assistants to run his business and ease his life as a CEO.

Max is a true open-source enthusiast, and he wants open source to win. Everything we discuss here is somewhere on GitHub, which I have happily linked throughout the interview.

Figma for Agents: Visualizing Tasks and Jobs with Agor

Before we start using Figma for Agents, coordinating them on canvas, we need to ask why we need coordination and orchestration in the first place.

Balancing Quality with Quantity: Messy DWHs

That’s where we started, with the challenge of messy data warehouse environments that most people find themselves in. I asked how he balances quality and quantity, aiming for high quality.

Max says that the new models, Opus 4.5 or 4.6², are not making many errors anymore and are very clever when they get the right context as above with all the database schemas of tables and data types, and even querying it with MCP. He says they almost run in self-serve mode, but he still prefers that users know what they are doing and can either read the generated code or verify the generated numbers on a dashboard or chat results.

But the setup is critical. With these three prerequisites, the agents handle almost all queries really well:

1. You need some preparation claude.md/ agents.md
2. Access to SQL (e.g., execute dbt)
3. Access to MCP or CLI for BI tools (e.g., Superset supports sup!, a CLI to interact with Superset)

The only problem, and always has been, is the messy structure and sources that most organizations have, growing from an initial small project into a certain stage. There are always obscure tables or strings, timestamps not aligned, or hidden information that is not encoded in code or written down. Or there’s the hidden knowledge, like that a certain table shouldn’t be used anymore or has bad data, which is known to the people using it but might not be to agents.

The Canvas in Which Your Agents Can Run: Automate Most CEO-stuff

When he recently saw the power of agentic coding, Max went all in and has been building the Figma for agents ever since. Something he can use to collaborate with agents within his company, instead of everyone running the same prompts locally and needing to sync with each other manually. That’s when Agor was born.

Agor stands for Ag: agent and Or: for orchestration. As the creator of Airflow and CEO of a data company, he knows exactly how a tool needs to improve his workflow. He also called it:

The goal is to automate most of the automatable CEO-stuff

Building an Internal Knowledge Base: Shared Canvas

Agor was Max’s answer to “how he uses AI beyond a research tool”, but doing data modeling, writing data pipelines, even legal or HR roles he added later to Agor, so you can give company-wide roles to agents that can be fed with dedicated documents and context, and triggered by any employee internally. In contrast, others see the jobs and avoid asking the same questions, reusing the output for new queries—Andrej Karpathy’s concept of an LLM-maintained shared team wiki—which builds an internal knowledge base.

Initially, when we first chatted, Agor had already changed how he worked as a CEO, but since then, Agor has gone even further. Agor can replace high-level tasks while still being very hands-on by working closely with the code via git worktrees (branch cards in Agor’s UI) and verifying the code in the PRs it produces. Max also added OpenClaw-like features around memory and identity via dedicated Markdown files, such as AGENTS.md, SOUL.md, MEMORY.md, so that Agor’s agents can learn from recent runs and carry a purpose and clear instructions. This led to role-based agents called Assistants (use agor-assistant as a template to build your own).

Inspired by OpenClaw’s agent loop, Assistants became first-class citizens, persistent AI companions with memory, identity, skills, and scheduled tasks integrated into Agor’s canvas, multiplayer workflows, and reachable directly from Slack, for example. Additionally, Agor adds features beyond OpenClaw, such as better multi-user support, RBAC, one-click, full session inspection, and many more.

Asked about the goal of Agor, Max said:

The initial premise was to remove DevOps and set up time for other members of the company. Instead of people needing to connect all the MCPs or CLIs to add API keys, set permissions, or integrate with Slack, the prompt window with the needed context is there and ready to start.

[!note] OpenClaw, what is it? And how does Agor compare?
OpenClaw (formerly ClawdBot) is an open-source agent framework built around a persistent agent loop: a serialized cycle that turns a message into actions, using file-based identity (SOUL.md) and layered memory (MEMORY.md). Agor’s Assistants adopt this pattern and extend it with multiplayer boards, RBAC, and canvas-level orchestration.

Visual and Spatial Memory

When you do a lot of agent work, it’s really hard to keep up with all of it. That’s where Agor’s visual and spatial overview really helps and is unique in its approach.

It brings the local and private session to a server, where everybody can see and work together on the same queries, and use the insights from other results, as dashboards are built for. So instead of keeping output locally, others can source the artifacts generated by agents, stored as Artifacts within Agor, ready to use by anyone, with no integration or deployment needed.

Or how Agor tracks its own spending across sessions:

[!note] Check the full Webinar about the Anatomy of the Internal Data Agent at Preset.

Such as an assistant needing access to all pipelines’ metadata as illustrated data stack.

Or the data needs an analytics agent to support, such as self-serve, the data team, and extras such as memory, skills, documentation, etc:

Context Layer: Back to Semantic Layers?

Max also believes that we are going back to the Semantic layer, or using it for AI as agents benefit from structured information - helping with the data model and SQL part, to make sure it’s correct, especially with the needs of AI agents and the persistent challenge of providing trustworthy self-service analytics.

With the shift of semantics outside of the BI tool, versioned, testable, portable, it’s a chance for better integration between business domain experts and data engineers. His thinking has evolved since he wrote the article, and Max told me:

I see two different semantics: the semantic layer and the YAML. There are the hard constraints — not every area needs that strictness — and then the softer ones with Markdown and Agentic Skills, good for 80-90% but with no guarantees.

AGENTS.md For Databases: Markdown Stored Inside the Database Itself

Based on that idea, Max created an experiment to bring the AGENTS.md convention into the database. DB-AGENTS reserves a dedicated schema and table, _agents._agents, that holds agent-oriented documentation at different scopes (global, domain, schema, table, and even column). You write the docs locally as markdown files with YAML frontmatter, and a small CLI (dba) deterministically syncs them into that table — since databases don’t let you drop files into them, the table becomes the file. Agents then query it at session start the same way they’d read an AGENTS.md, making it a natural companion to INFORMATION_SCHEMA: one holds structure, the other holds meaning. Max calls it a “soft semantic layer”, which maps directly onto the hard-vs-soft split he described above. Check out the repo at db-agents.

With context being key for agents to understand what we humans know, Agor also added a context layer called knowledge. Agor Knowledge acts as a central place where humans and agents can store, organize, connect, and find the context that makes work compound over time, with Slack quickly becoming the main interface to many of the team’s agents.

How Do We Sandbox Agents for Safe Workflows (Okta for Agents)

Another big topic is security when agents have so much access to powerful CLIs, sometimes root access to systems or databases containing private keys, or just downloading random skills from the internet that may contain hidden secret messages.

Max coined the idea of Okta for Agents, which I found super interesting, and something I believe will become ever more important if we want to find a healthy way of working with agents in enterprises or with sensitive data. Okta for Agents means working around identity, scoped delegated permissions, leases, and audit logs.

When asked how he’s managing security, verifying what Agor or the agents are doing, Max responded:

I let the workers run in god mode³, but using dedicated environments/sandboxes, hooked to a dedicated git worktree repo, it can run autonomously and solve problems on an initial prompt, visualized in a shared canvas style.

I asked how he sees Okta for Agents being implemented. We desperately need it, he said, granting agents permissions like impersonation. Delegating the permission is an OAuth. With the roles, we can scope permissions strongly. E.g., the sales agent only has access to sales documents.

When asked at what level to integrate the Okta security layer, Max said it hasn’t been solved yet. Still, he sees it as the same question: whether we have 50 agents or 50 users who use a platform, both need a security layer.

Likewise, Max shared:

I trust agents the same way as I would an employee.

[!note] What does “Okta for Agents” mean in more details?
Okta is the identity layer companies put in front of their tools: it authenticates who you are, decides which systems you can open, and logs what you did. Max’s point is that agents need the same layer. E.g. Clawdbot, when he tried it, was effectively a DIY IAM manager for agents, config hell and all.

The twist is that it adds a whole new dimension to RBAC. It’s not just “the bot gets an email account” — it’s “the bot gets an email account, but can only read mine, and via MCP rather than as a real user.” Not so different from onboarding a human personal assistant, except this assistant can help with nearly everything, so the blast radius is much bigger.

Max’s own example: he saw a 1Password skill and immediately backed off then reconsidered, wondering whether the bot should have its own 1Password account with only safe credentials shared into it. Which is exactly the problem: you can be strict on paper, but the moment the agent has your email, Slack, and calendar, it can leak private things all day. (Full discussion at this post)

[!warning] Security is critical. Here are examples when it’s gone bad
How I Dropped Our Production Database and Now Pay 10% More for AWS, or another one, or when 13-hour AWS outage reportedly caused by Amazon’s own AI tools. Or also just hacks by getting attacked via GitHub PRs or How secret instructions injected into skills.

Declarative and Non-deterministic Outcomes?

Related to security is the deterministic, repeatable behavior of data sets with the same input. Agents are the opposite: probabilistic. I was curious to hear from Max, who initially defined the functional data engineering paradigm for deterministic and idempotent batch data processing, what he thinks about the non-deterministic outcomes of agents, specifically with large language models.

Max said, regarding declarative definitions, that he finds a claude.md is usually sufficient for most tasks that have a git repo, more context, and an issue or PR to work with, given the initial prompts come from users who know what they are doing.

Regarding reliability, Max thinks about using good methodology references. Agents get it and understand it. E.g., data modeling practices such as Kimball are still valid, or the approach shared by him with Entity-Centric Data Modeling (ECM), he says, and when prompted to model in those patterns, agents follow them well (either via research or provided).

The other part is that some non-dangerous work can have vibe data pipelines, and there’s no danger. And there are cognitive-depth tasks, such as a complex Spark cluster, where you can’t just debug quickly with large data sets.

Also, the field varies: not every area is getting agentic-piled as fast. E.g., platform demands go through the roof (see GitHub outages), so we have 10-20x the platform needs, but at the same time, the work is critical to be correct. So it depends.

Future of Agentic Workload, and Canvas Development in Teams

When asked about how Agor has changed how they at Preset develop products (if at all?), or made them more effective, Max said:

There are more agents than humans nowadays. Everyone has a Claude Max plan, and agents handle almost all code writing.

And on a personal level:

I haven’t written a function by hand for a long time, and I might not anymore — except when I feel nostalgic.

He also thinks that the Yap-to-Ship Ratio, a metric that he announced half-jokingly on LinkedIn, describing people’s velocity by just getting stuff done without involving others at every step, will be very low-yap for 10x engineers, as they solve the problem and ship a solution without much back and forth.

They deploy it somewhere for others to use, not only for human consumption, but as a solution or CLI that other agents can use to discover further and solve their problems. A high Yap-to-Ship ratio would mean lots of human interaction, which is the clear new bottleneck.

As human code review gets bottlenecked, I asked how he does the review. He said he uses Codex with sub-agents to review, ensuring everything is DRY (Don’t Repeat Yourself) and that all expected callbacks are made.

You can also ask the operator assistant agents if you are not sure whether an implementation is correct.

Amdahl’s Law: Can’t Go Faster if not End-to-end

One bottleneck is still Amdahl’s Law. We can speed up tooling by using extremely fast agents, but unless the end-to-end workload is sped up, we only increase by a 2-3x factor, not 10 or 100 as any one tool does. This also overlaps with Max’s Yap-to-Ship ratio: if PRs need the human in the loop to review many of them, the overall speed at which we build is not faster.

Another side effect is that it takes a lot of context switching. Max said he has ten active sessions in Agor. He is good at context switching (maybe also learned through recent Agor workflow? 🙂).

Predictions for 2026

It’s hard to predict the future with AI, but Max took a stab and shared his predictions for 2026 and categorized them into wired and tired:

Max’s AI Setup for Data Engineering Work and Managing His Company

We end this interview with Max’s setup for working with agents, since we didn’t have time to go into full details on the call. I’m sharing the one he shared four months ago. I’m sure it changes almost daily. Still, it helps us get a good overview of his software engineering stack for the team at Preset, as well as his personal local computer stack.

For software engineering and data engineering:

• Preset team instance of Agor behind VPN with a dozen boards, boards are mostly repo-oriented. Full Unix impersonation, backed by PostgreSQL.
• doing most of my work on board with the agor-openclaw framework, agent is pushing projects across a kanban-type layout: tons of new automation there. Agent checks on agents, prompts them, moves worktrees to “needs human review” zone if/when needed
• coding workflow is Opus 4.6 as a planner, Sonnet 4.5 / Opus 4.6 for most coding, Codex 5.3 as the reviewer (god it’s so good)
• data engineering stuff: dbt/airflow repo + Superset MCP, superset-sup
• “Command center” is agor-openclaw on Opus 4.6, monitors other agents, intricate HEARTBEAT.md with pseudocode to make a decision for coding project (worktree) on that board

And his new “Personal assistant” local instance of Agor (brand new/sensitive):

• Beefy Mac Studio at home
• connected to “productivity” tools (google-workspace-mcp)
• connected to Slack through a semi-homegrown skill — read-only for now, mostly to summarize activity
• connected to Notion MCP
• connected to “contracts” repo, where I sync with Google Drive for all Preset contracts
• connecting to Hubspot soon
• goal is to automate most of the automatable CEO-stuff as discussed above

Coming up

We’ve learned how to use Figma for agents with Agor and to collaboratively work as a team, using shared prompts and creating artifacts. We’ve seen how Okta for Agents is needed but really hard to implement, and how the future of AI is mostly about context and how to integrate it well. Plus, we learned how Max automates many tasks as a CEO with dedicated AI assistants and still produces low-level code with the same assistants for both his personal and company-wide needs.

I hope you enjoyed this fourth interview with Max. Huge thanks to Max for taking the time to speak with me (twice!) and for sharing his experience with all of us. Follow him on LinkedIn, GitHub, or on Preset Blog, where he shares his distilled thoughts on the ecosystem, and obviously, if you want to know more about Agor, check it out at Agor GitHub repo.

Max shares a lot of his ideas and thoughts online. Here are some further articles and interviews to read/watch:

• AI Enablement Engineer: The Highest-Leverage Role in Tech
• Check the Joe Reis interview where Max shares more on the discussion From Airflow to AI Agents: On Building Agor and Running a Company with AI Agents

More interviews are coming out, so please share feedback, questions you might want to ask, or your experience working with AI in the data space. We’re all in this together, figuring it all out.

Full article published at MotherDuck.com - written as part of my services

As I love books and recommendations by others, I want to share them in a collective format so others can read gems of books, as I think books are still the best way to read new information these days. Even more with the fast pace we are going at, as books are well structured and made for the long term.

Book Recommendations I’ve Read

Over the years, I have made recommendations in Writer’s Room, my newsletter, on my now page or in person. Here I have collected and curated them over the years including some comments. I hope you enjoy.

2026

• Poems & Prayers (Matthew McConaughey): Another one read by Matthew himself. It’s about life and its rhymes. He said life is like rhymes as he likes them, sounds good, and prayer because of the meaning. Together applied to life, it’s like a dace to the day. He also says he has a rhythm in his head when he wakes up, and then he dances to it somehow during the day, and life fits into the song, making his day like a dance. Also, people say he is so laid back, but it’s because he has a plan, he does not rush, he takes time. So that’s on purpose. Something I also actively try to do in my life. Not yet finished.
• Maintenance of Everything (Stewart Brand): This is just so good, it tells the story of how maintenance is everything if you rely on a boat on the sea, for example. Making the product easy to repair so you can fix it even in a rough sea with limited tools. It’s an analogy you can apply to every part of life, and the way the story is told is so great. I couldn’t stop listening to the audiobook. I really recommend it to everyone; I’m sure it will give you lots of insights into your current work.
• Big Time (Laura Vanderkam): Started with the goal of getting more out of my time. So far, I’ve learned that time management is essentially about controlling time, but it’s not the only way. And that “Shifting Left” is not only used in data engineering, Laura describes doing tasks long before they are due as shifting left too. Good reminder of how to use time better, nothing revolutionary, but lots of real-life examples from her and her clients.
• The Writing Life (Annie Dillard): A poetic way of what it means to be a writer. Great storytelling. Learning how to write with a family as part of her stories.

2025

• Novelist as a Vocation (Haruki Murakami): Such an amazing book on writing. It really reminded me of my writing style, as he translates words and tries to write beautifully for the sake of beauty, not to impress the reader. No timeline. It takes as long as it takes until the quality is good. And he likes a good challenge to keep him engaged, e.g., when he goes abroad to start from scratch. His style is so unique and amazing. It’s a joy to listen to.
• Greenlights (Matthew McConaughey): This book was so real. Matthew reads and shares from his 35 years of journaling and life. So inspiring and [[Writing from The heart|straight from the heart]]. It was interesting and motivating to hear from a Hollywood actor and celebrity what truly matters: family and keep livin. Alright, aaaalright, alriiight. → I listened to the audiobook, and his Texan accent was the best. He was rhyming, slowing down, calming down, getting loud, all of it. It was like sitting next to him and listening to your dad or uncle.
• Consider This (Chuck Palahniuk): The author of Fight Club shares how to write if you were his student. A masterclass if you want to enhance your writing.
• You Can Negotiate Anything (Herb Cohen): Interesting perspective that you can negotiate even at places you’d never thought of, even at a shopping mall. Always keep in mind what the other’s objective is. There are always three elements that are part of a negotiation: information, time, and power.
• The Bible: I also started to read the Bible, the most sold book on earth. I wanted to understand what’s in there. Obviously, it’s so massive, and it’s actually not one book, but a collection of many. I get a similar sensation to one of my favorite books The Daily Stoic (366 Meditations). It’s like a daily meditation. While Ryan Holiday’s book helps me feel calmer, the Bible nourishes my love and care for others.
• Stolen Focus: Why You Can’t Pay Attention (Johann Hari): This book inspired me to write an article about flow myself: Finding Flow. Super insightful and probably will make you want to ban social media by the government.
• Useful Not True (Derek Sivers): Derek is the master of stripping away. This book has so much wisdom in short one-page stories from Derek’s life. It’s a gem. Perfect to re-read, over and over. It’s next to my sleeping place to just grab.
• Tao Te Ching (Lao Tzu): Dao and Daoism: Men - earth - universe - dao. If you don’t want to read the bible, this is a none religious way of getting solitude and compassion.

2024

• The Good Work (Paul Millerd): A continuation of the famous Pathless Path, not as good, but OK. Please read it if you haven’t yet.
• The Extended Mind (Annie Murphy Paul): A powerful book. I started reading it again.
• Do The Work (Steven Pressfield): It’s not the best book, maybe also because I’ve heard or read lots of the content already elsewhere, but I appreciated that it was short and that he didn’t try to extend it to 300 pages.

2023

• Slow Productivity (Cal Newport): This is a fantastic book. It showcases why we should quit the race of everyday life and slow down for a compounding effect instead of overnight success. The same is true for money: investing long term instead of gambling in a casino. He describes that instead of pseudo productivity, which is the norm these days, where we try to be as busy as possible to showcase we are doing something, we should try to be as productive as possible.
• Elon Musk (Walter Isaacson): It’s very long, but I loved every page (or word as I listened to it on Audible). It is inspiring. It shows how an exceptional, hard-working Musk is doing everything for humanity in an unhealthy way. He is also, to an extent, sick. He has his dark sides that just come out sometimes, which are also a sign of his mental state that he got from his dad, and also from his hard work and pressure. He is thriving in chaos. Whenever there is a calm time, he will do something new as he can’t stand the status quo.
• How Will You Measure Your Life? (Clayton M. Christensen): A good book if you want to know more than just work and how you’ll measure it. This book was reassuring and strengthened many things I already knew. It reminded me that finding your principles is the key. Follow them and align your life so you have a happy life. It was also helpful for me as a dad and family member to pass on the same principles and values to my kids. Be intentional about your values. It’s hard to find them. They won’t be sent to you. You need to make them. But be aware that it is a process, not an event.
• The Good Enough Job (Simone Stolzoff): A compelling book that challenges our conventional thinking about work. Instead of idolizing our jobs or incessantly chasing a better one, Stolzoff advocates for finding satisfaction in a “good enough” job. His ideas offer a refreshing contrast to the pervasive Instagram-era narrative that equates career success with personal fulfillment. A highly recommended read for anyone feeling pressured by the modern-day cult of work.
• The Daily Dad (Ryan Holiday): This book is a sequel to one of my all-time favorite reads, The Daily Stoic. In the same tradition of offering daily philosophical advice, this book focuses on the challenges and rewards of parenthood. TODO
• The Extended Mind (Annie Murphy Paul): An exploration of the intriguing ways our environment influences our thinking processes. Discover the surprising ways in which experts think beyond their brains, how harder thinking often leads to fewer results and the controversy over brain-training games and smart pills. Explore how we can use tools beyond the brain, such as the Body Scan technique and meditation, to tap into our intuition and sensations, and learn about the significance of the amygdala in our responses to stress.
• All the Wrong Moves (Sasha Chapin): A captivating narrative where the world of chess serves as a backdrop for introspection and self-discovery. A story that shows how the love of chess can fully dominate one’s life, as it’s the most beautiful and worst thing in life at the same time.

2022

• Atomic Habits (James Clear): The main argument: If you want to add a new habit, chain it to an existing one to make it stick.
• Building a Second Brain (Tiago Forte): I knew most of it from his articles, podcasts, etc., but now it’s also available as a book.
• Getting Things Done (David Allen): This book is life-changing if you apply its principles correctly to your life.
• How to Take Smart Notes (Sönke Ahrens): The base for a Second Brain and where Sönke reveals how our brain is wired and how we can implement a note-taking style that supports our brain, mainly with the method called Zettelkasten.
• The Pragmatic Programmer (David Thomas & Andrew Hunt): Although a lot was clear, summarizing it and putting it together as one piece, plus hearing it from two professionals, was very helpful and suitable for applying to my work.
• On Writing (Stephen King): He gives deep insights into the life of a successful writer.

2021

• The 5 Love Languages (Gary Chapman): Really eye opening, as sometimes we want to show love to our spouse, but becausewe give attention, or buy a gift, but if it doesn’t match the “love language” of the reciever, it won’t arrive properly. So knowing each others love language, and communicate in that, is huge for any relationship.

2020

• The Courage to Be Disliked (Ichiro Kishimi, Fumitake Koga): Great book to learn new psychology, especially the Alfred Adler theories of individual psychology and Adlerian therapy. And learn about Interpersonal Relationship. This books makes you stronger.
• Your Music and People (Derek Sivers): Fantastic. It tells the story about how to sell your music, which is an analogy to every business.
• Mindwise: How We Understand What Others Think, Believe, Feel, and Want (Nicholas Epley): Approaching our mind from an angle you probably haven’t thought of. Very insightful.

2019

• Thirst: A Story of Redemption, Compassion, and a Mission to Bring Clean Water to the World (Scott Harrison): This book is truly heart-touching and lets you rethink today’s world and donations. Very well-written book.
• It Doesn’t Have to Be Crazy at Work (Jason Fried & David Heinemeier Hansson): Talks about the crazy work hours usually suggested or heard at places like Google, Amazon, etc. to be able to achieve something. They tell you how you can tackle it otherwise.

2018

• Principles: Life and Work (Ray Dalio): I recommend this to everyone out there. The first chapters where he talks about himself you can skip, but the rest is worth reading or listening to. TODO
• Unshakeable: Your Financial Freedom Playbook (Tony Robbins): If you want to save some money in the long run, please read it. It is outstanding and straight to the point!
• The Tao of Seneca: Practical Letters from a Stoic Master (Seneca): Puts me back to the essentials of life whenever I feel all over the place.
• Tribe of Mentors: Short Life Advice from the Best in the World (Tim Ferriss): Entirely recommendable.

General Recommended Books

These are some of my all-time favorite books that I recommend to everyone.

Top Books

• The Daily Stoic (366 Meditations) (Ryan Holiday): One of my all-time favorite books. Daily meditations that help me feel calmer and more grounded.
• Hell Yeah or No (Derek Sivers): Anything by Derek Sivers is worth reading. His writing is stripped to the essentials and packed with wisdom.
• How to Live (Derek Sivers): The second book by Derek on top list. He writes 27 conflicting, and short chapters on how to live life. He has done so much, that he writes almost each from experience. As it’s short, each chapter can give you a nudge and insight into your own life, and what we should strive, and more importantly, what not. Be aware, every unnecessary word is removed and stripped to the bare essentials. This means you can’t read more than two chapters in one go as it’s that condensed and makes you think a lot.

Business

• It Doesn’t Have to Be Crazy at Work (Jason Fried & David Heinemeier Hansson): Challenges the norm of crazy work hours and shows you can achieve more with less chaos.
• Deep Work (Cal Newport): Essential reading on focused work in a distracted world.

Self-Help & Life Philosophy

• Thinking, Fast and Slow (Daniel Kahneman): A deep dive into how our minds work, exploring System 1 and System 2 thinking.
• The Pathless Path (Paul Millerd): Challenges the “default path” (going to school, work, mary, and have a family) and following your instict. Unconventional way. The new 4 hour work week book. One that I recommend the most lately. I wrote about Finding my Pathless Path.
• Four Thousand Weeks: Time Management for Mortals (Oliver Burkeman): A refreshing take on time management that embraces our mortality rather than fighting it.
• Principles (Ray Dalio): Life and work principles from one of the world’s most successful investors.
• Emotional Intelligence (Daniel Goleman): Understanding and managing emotions for better relationships and decisions.

⭐ 5-Star Picks (GoodReads)

• On Writing Well (William Zinsser): The classic guide to writing nonfiction. Essential for clear writing.
• So Good They Can’t Ignore You (Cal Newport): Why skills trump passion in the quest for work you love.
• Steal Like an Artist, Show Your Work!, & Keep Going (Austin Kleon): Probably the best book on creativity and if you are a creative yourself. It’s really helped me sharing more of the process and how creative works. It’s a trilogy, I listened the audiobook, which you get three for one, fully recommend it.
• Still Writing (Dani Shapiro): Honest reflections on the creative life.
• The Psychology of Money (Morgan Housel): Best book on understanding money and behavior.
• Moonwalking with Einstein (Joshua Foer): Fascinating journey into memory and how it works.
• Open (Andre Agassi): Raw, honest autobiography. Surprisingly insightful about life and finding yourself.
• Anything You Want (Derek Sivers): 40 lessons for entrepreneurs. Short, punchy, full of wisdom.
• The Subtle Art of Not Giving a F*ck (Mark Manson): Counterintuitive approach to living a good life.
• The Alchemist (Paulo Coelho): Beautiful fable about following your dreams.
• Hooked (Nir Eyal): How to build habit-forming products. Essential for product designers.
• The Startup Way (Eric Ries): How modern companies use entrepreneurial management.
• The 4-Hour Work Week (Tim Ferriss): A game changer that questions conventional work structures.

Further Reads

• Reading Books for a Happy Life: My thoughts on why reading books matters.
• Audiobooks: I listen to most of my books as audiobooks. It works better for me and my brain, as I read all day long already. Books read by the authors on Audible are just another level. It feels like listening to them in person. Although a physical book has advantages, audiobooks are just something different. No wonder they are ever-growing.
• Writer’s Room: Where I write and update on latest insight in my design process.

Other Recommendations

• Derek Sivers book notes

Last updated: December 30, 2025

The Act of Creation

The act of creation is an outlet. It gives joy to us when we create something out of nothing, we block out anxiety or boredom.

Creating should be done like:

• a kid in mind: effortless, exploring your thoughts, and seeing where it leaves you, as Picasso said: He wouldn’t have bothered to start a painting if he knew the outcome.
• a play¹: flawless and free.
• calm like the water: flowing wherever your mind is going.
• to fulfill yourself: by portraying your never-resting thoughts. Share it with the world. Create with joy and love in mind.
• follow your train of thought: see where it leads you. You start somewhere, on an unfinished emotion, a task, or anything to ponder on.

Creation, and especially writing, can help resolve the unresolved.

Creation to Make a Difference: Joy and Laughter

Creating is also about making a difference. If nothing changes in you, the people, not enough value was added to your creation. Any creation should trigger something. But foremost, the process of creation should change something in you, some release, a good feeling, it must feel right. If it does nothing, an empty feeling, the creation might not be there yet.

Create with love and empathy in mind. Liking someone is one thing, but giving love, caring for someone, and integrating that into your work is the ultimate way to create.

Create joy and laughter. Speak the language of people, make them happy by creating enjoyment.

Following Our Emotions without Boundaries

When we follow our emotions without any boundaries, following our instincts, we get into a state of deep focus and deep concentration where all thoughts can be resolved.

Creation is our inner [[Gut Feeling|Instinct]]. If we follow and nourish it, great things can arise from it. Treat it as an outlet for your body and mind. Go with the flow, follow along. Create for the sake of creation.

Learn for Life, Follow Your Own Uniqueness

The act of creation is following your own life rhythm, for example, the [[Pathless Path]]. It is the ultimate form of connecting with yourself.

All creation is unique, unique specifically to you as the creator, but also (hopefully) to the reader, viewer, or consumer of your work. Your creations live long after your death; others will connect and create new works from them. It’s giving joy after you’re gone.

Learn for Life, as I like to say, has been my earliest mantra when starting any of my creations shared online. The common definition of success is reaching a desired outcome. The Scientist’s definition of success is: if you learn something new, you haven’t failed, as Anne-Laure Le Cunff says on How to Design Tiny Experiments Like a Scientist.

For example, a scientist defines it like this: if you say you want to write more, instead of just writing, you define ‘I will write for at least one month, or at least 10 articles. And then you don’t stop before that. Success is the experience, not the outcome.

The Outcome of Creation

The act of creation is a slow process that needs time and experience. A slow life, an [[The Ordinary (Boring) Life|ordinary, boring life]] even, to focus on the details, using the craftsmanship refined over the years, and retrieving joy from the process of creation.

What matters is the quality and outcome we are proud to release to the world.

Gifts Shared with the World

You create for yourself with no return in mind, you just share it as a gift for the world, for anyone to consume.

Creation thought of as a gift is an easy way for you to create without the burden of pleasing people, as it’s take-it-or-leave-it, like a gift. No strings attached, just making gifts.

A Curious Mind is Mostly a Subconscious Mind

When we create, most of our subconscious is driving the thought. Just let it cruise and see what the outcome is. Use the conscious mind for fixing errors and making sure the sentences make sense later, but don’t start with it.

The Spark of Joy Philosophy

Different flairs for designs to add ease or user delight to your creation. Similar to the “Spark Joy Philosophy”, which is very fitting when creating for the enjoyment of the reader.

To me, creation sparks joy for myself, it’s the outlet for me to release and organize my thoughts. It avoids the [[shallow happiness]] that I get from social media, binge-watching Netflix, or other brainless activities on the phone or TV. Sure, there’s time for that too, to calm down, but if we are not careful, the algorithms will take over and we default to always choosing the easy choice.

The Reward is Long-term

Creating is hard, there’s friction, it does not directly work, but the reward is long-term, with [[deep happiness]] and a deep flow state: “It’s magical that they just tried to be there”, which is the highest form of happiness artists get (see The Artist’s Way) as we discovered in Finding Flow, with escaping digital distractions through deep work and slow living.

I hope you find your outlet for creation, an act you can partake in to develop happiness and joy for yourself, getting into that deep flow and having a slow process where many small gifts of creation can be shared with the world.

This article was created with the inspiration of reading How to Live by Derek Sivers in Cavallino, Italy.

For text editing, we have Vim motions that help us navigate a text document with 1000s of shortcuts, but because there is a grammar, we do not need to remember them all, but learn the structure of the grammar and combine them. But what if you work in data? What if we could have the same for data, a grammar for data engineering, or a language that defines it?

Expressing our needs declaratively and decisively? Also, expressing it in a way that leads to reproducible outcomes, or works with multiple parts and execution engines already out there. This is what we will discuss in this article. How existing tooling, such as Ibis, provides some capabilities, and how xorq extends them by adding full lineage and transparency for humans, with included executable memory for useful tabular data, all manifested in a single git repository.

Expressions for Data Engineering Workloads

Having a grammar for data engineering means we can express the workloads in a declarative manner, and then be sure we can deterministically reproduce and apply that exact definition.

It’s similar to the concept of a Declarative Data Stack I introduced a while back, but it gives the stack not only configurations but also a language with in-built manifestation and execution engines.

In the above image, we see:

1. How to express (write) our transformations and business logic. It’s the context of every ML or DE pipeline.
2. We can build the expression into a manifest that has a unique hash, runs input validations, tracks lineage, creates a deterministic cache, and produces a human-readable expr.yaml you can diff and review in a PR.
3. Lastly, we can execute it in any execution engine with the same manifest.

This is hugely powerful and separates the concerns of defining logic, verification in the manifest step, and execution as a composable data stack, as Wes McKinney called it, with multi-compute engine possibilities.

How the DE Language Works: Different Expression Types

Every grammar starts with nouns, and here the noun is the source, a node that holds data but carries no transformation yet. It might be an in-memory table, a registered connection to a warehouse, or just a lazy pointer to a file on disk that hasn’t been read. They’re simply referenced, the way a noun refers to a thing before any verb acts on it.

The verbs in our language are transforms such as filter, select, mutate, aggregate, join, order, limit. Each one takes a source (or another transformed expression) and returns a new, immutable expression. You do not mutate anything before it, only describe what should happen next.

Looking at a definition such as .filter(...).aggregate(...).mutate(...), we can see this as a sentence. The moment a verb is applied, the expression stops being a plain noun and becomes a statement, a description of “data plus what should happen to it.” But the sentence isn’t spoken yet, it stays inert, fully composed but unexecuted, until something finally asks it to run. That’s the deferred part of the grammar: writing the sentence and saying it out loud are two different acts.

There’s a third part of speech worth naming: the template. Instead of writing a sentence about a specific noun, you can write one about a noun’s shape, a schema with no rows behind it. A template says “given something with a column of this type, here is what I’ll do to it,” and only later gets bound to an actual source, at which point the placeholder resolves and it becomes an ordinary statement again.

And we have modifiers that ride alongside a statement without changing what it computes. They’re small tags of metadata that say “this expression also represents a fitted model” or “this is a saved reference to something else.” It’s like a footnote with additional metadata that doesn’t change the surface meaning, but adds context for later use.

This analogy makes the grammar compose the same way regardless of which engine eventually executes it. There are more parts, but with just these four, noun, verb, template, modifier, you can read (and write) arbitrarily complex data pipelines the same way learning a handful of verb-and-object combinations in a text editor lets you compose arbitrarily complex edits.

[!tip] Avoids building “Inner-Platform Effect” with repeated tools
With this grammar, we can avoid repeatedly implementing the same logic we already have, but manifest and express our logic once, and reuse it with different execution engines, exactly what Ibis and xorq allow. Similar to what the inner-platform effect means for software best practices.

Why a Grammar is Really Good for LLMs

Having a grammar is really good for LLMs, too. It helps them first to declare data artifacts and second to execute them reproducibly.

On top, expressions can be LLM-agnostic, and we can interchange the LLMs we use just with an expression. Also, the chart is just an expression, or the data catalog and the metrics.

Model Once, Represent Everywhere: Expressing the Full Data Stack with a Single Expression

Like UDA (Unified Data Architecture) from Netflix, we define our expressions once and represent them everywhere. Netflix built UDA to solve duplicated models, inconsistent terminology, and siloed systems, where the same concept like ‘actor’ or ‘movie’ gets modeled differently across teams, with no shared foundation. Their answer was a full knowledge graph with a metamodel, making the conceptual model part of the actual control plane.

Not everyone needs Netflix-scale tooling, though. For a code-first approach, xorq gives you the same core principle: define once, execute anywhere by writing a declarative Ibis expression, serializing them as content-addressed YAML artifacts, and running against any supported engine, fully reproducible.

The difference worth noting: UDA is a semantic layer defining what data means across systems. Xorq is a computational layer defining what transformations do across engines. Both reject the same anti-pattern of re-implementing the same logic for every system.

Entering Xorq: The Horizontal Data Architecture

Xorq is an executable memory system for tabular data that works horizontally across your data stack, supporting everything from discovery with a catalog to defining transformation logic to modeling.

It has declarative transformation (Pandas style), and you can build ML pipelines and prepare data with its semantics in a single stack that is not vertically integrated, but horizontally integrated, giving your agents a catalog of executable pipelines and turning short-lived agent work such as wrangling scripts, sklearn pipelines, ad-hoc tables into durable, composable, executable artifacts that any future agent or human can discover, reproduce, and reuse.

The horizontal data stack shows what Xorq brings to the table. Xorq’s origins started from a git-native semantic layer, for data analysts out of college, to build semantic models for a living, to make their lives easier.

From point-and-click tools, dragging tables and drawing joins manually, only to add more reporting tools on top to create pixel-perfect reports. Also performance-wise, it didn’t scale, meaning we needed cubes to make it faster, adding another layer of complexity.

And there was no lineage that shows from source to dashboard. The question asked was: “what if we could do this end-to-end data engineering workflow locally?”. This is what the horizontal data stack and xorq are providing.

To add semantic layer capabilities, Julien Hurault and Hussain built the Boring Semantic Layer + the Xorq catalog, providing a semantic model you define in Python, check into git, and query from the CLI.

Compressing Logic into a Single Executable

Compression of a full data stack into a single executable is hard, but xorq tries exactly this with the help of Ibis, git, uv, and DataFusion.

The design choices of xorq showcase even better what it is, and what they enable:

• Ibis as expression layer (v9.5.0+, partial): Declarative dataframe expressions compiled to multiple backends (xorq supports a subset of the Ibis API, not the full surface)
• Git for state and storage: The catalog is a git repo of entries with git-annex support for large files
• uv for reproducible environments: Each entry ships with a wheel and pinned requirements.txt.
• DataFusion for embedded compute: Pipelines execute in-process with SQL and UDFs

Composable Data Engines

Another big advantage of expressions and having a grammar for data engineering is easily switching between backends, with no change to the transformation or business logic. It’s just defining the backend from Apache Arrow Flight to DuckDB or any other engine.

We write the definitions and express our tabular data and computations. The engine, in this case xorq, can build it into a manifest file that is deterministic and hashed.

Xorq uses Ibis as the expression layer for single-backend logic, then builds the cross-engine expression tree into a serialized YAML artifact. When moving data between backends, xorq transfers Apache Arrow RecordBatch streams between them—each backend acts as a RecordBatch transducer. No CSV serialization, no JSON encoding needed. This makes backend switching fast and memory-efficient. Write declarative Ibis expressions that run like a tool—xorq extends Ibis with caching, multi-engine execution, and UDFs.

Here’s an example of using DuckDB and Postgres in conjunction:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

import xorq.api as xo

# Connect to engines
pg = xo.postgres.connect_env()
db = xo.duckdb.connect()

# Load data from different sources
batting = pg.table("batting")
awards = xo.examples.awards_players.fetch(backend=db)

# Filter in respective engines
recent = batting.filter(batting.yearID == 2015)
nl_awards = awards.filter(awards.lgID == "NL")

# Move data to postgres for join
result = recent.join(
    nl_awards.into_backend(pg),
    ["playerID"]
)

result.execute()

Move data between different engines within a single expression using into_backend(), here Postgres and DuckDB

You can see how easily you choose your most optimized execution engine, whether in the above example choosing DuckDB for filtering recent batting and using Postgres to filter NL (National League) awards, and joining the two with the Postgres engine.

Engines supported by xorq as of now, with the ability to move data between them, are (check Supported backends for the latest):

• Embedded: DataFusion, DuckDB, SQLite, Pandas
• Warehouses: Snowflake, Databricks, Trino, Postgres
• Lakehouse: PyIceberg
• Arrow Flight: GizmoSQL (DuckDB over Arrow Flight SQL)

Cross-Engine Expression Tree

With different engines supported, we can use the compressed single executable logic across engines. We can build expression graphs before executing them, which works like this, with one expression, many engines:

1
2

expr = penguins.into_backend(xo.sqlite.connect())
expr.ls.backends

The output of building a cross-engine expression is a directory containing your serialized pipeline with a unique hash identifying each build and its artifacts and expressions. When executed, the output is the resulting object or data.

And the expressions are tools, Arrow is the pipe. E.g., a Unix pipe streams text between small programs. Xorq pipes Arrow streams between expressions: unix : programs :: xorq : arrow-transforms

That executes like this:

1
2

In [6]: expr.to_pyarrow_batches()
Out[6]: <pyarrow.lib.RecordBatchReader at 0x15dc3f570>

This is quite short and potentially abstract to understand when never used, but we will go into more examples and details in another article.

A Shared Language for Data

This article introduces a new way of describing data transformations for machine learning or data engineering pipelines in a direct and simple way that works locally with any execution engine, without changing the code itself.

It’s a good place if you need a trusted harness for a data engineering persona. We can define once and use it with the engine that works best for your workload and data engineering environment.

We had a look at how we write -> manifest -> execute with xorq, its advantages, and why you might use it for modeling once and representing everywhere. By adding AI agents to the mix, which help us pull the right lever, instead of bigger, more expensive models or more tokens, we improve accuracy with more semantic understanding, with a grammar the model can learn and apply, even pre-manifest before execution, and run them deterministically every time. This is a huge addition to working just with agentic Skills files that are free-form Markdown and pull data all over, or are not defined precisely enough. It’s all about having high-quality context in the right format, with a clear definition where humans and AI agents can interchange and help each other.

There’s a lot more to come, with showcasing the horizontal data stack and the use cases it supports, how we build expressions versus running computations, and how data catalogs are integrated into the picture, too.

–

Check out xorq code and star it on GitHub, or read more behind the scenes at Xorq documentation.

They also have a macOS desktop app coming up that does it all in one unified app, geared towards non-technical users. Join the waitlist for that.

Full article published at xorq.dev - written as part of my services

The danger isn’t only a crash or an error message, but a wrong number that didn’t break. It might be a clean query, but it introduces duplicated rows.

In this article, we go through the three levels of AI agents in data engineering, how to structure projects so the AI delivers its best outcomes, and how dedicated agents with a deterministic core help us build higher-quality pipelines — ones we can actually trust. And we look at a practical example of how it works with a blast radius analysis.

The Three Levels of AI Agents in Data Engineering

Why should we use agents for data engineering? And at what levels can agents help us productively? As LLMs will always have some error tolerance, as humans do too, we need a way to be more confident in producing the code.

Chat-phase, Autonomous and Dedicated Tooling

There are different levels of confidence and levels on which the agents can help us.

1. The initial chat-phase: the development where we prompt Claude or ChatGPT. The model tries to understand the context based on what it has access to. It takes a decent amount of tokens, as it needs to scan everything from scratch.
2. The autonomous approach, where Claude Code or Codex also have access to the tools humans have, mostly the CLI on the terminal, making it possible to query Postgres with psql or read from S3 or Parquet with DuckDB to verify queries and data. A much higher quality outcome.
3. Dedicated agents for the task at hand. E.g., for data, the tools know dbt or know how to transpile SQL code deterministically, meaning not from training data only, but with an actual tool that does it much faster and more reliably. Built-in checks and features a “general” agent can’t provide.

Ideally, we’d want to always use the dedicated tools, but there isn’t always one.

Where in the DE Lifecycle Each Level Actually Helps

BI Dashboards vs. Plumbing the Data Pipelines, or Creating Source Ingestions, or Maintaining? For data engineering, the question is not only if there is dedicated agent tooling, but also on what part of the data engineering lifecycle AI agents can help data engineers and analysts the most, and potentially even domain experts?

The lifecycle contains the ingestion part, ETL, or understanding the business in great detail, or is it just to visualize the result? Or should it cover maintenance in case of overnight ETL errors, or the full data lifecycle?

In general, before we go into more details later, agents can help us on the full cycle, but it always depends on who you are and what role you play. Building from scratch with no knowledge or seniority is dangerous. Why? Because they can’t verify if the produced code is correct. Okay for a side project or a proof of concept, but not for actual production.

What’s the Engineering Discipline for Working with AI?

There’s also a part that is less technical, a way of guiding the agents in the right direction. Especially if we want to safely use it in large projects or organizations, we can’t just let it run without guidance.

For that we need:

1. clear project structure in which the agents can flourish. The more is given, the fewer tokens are used for this work, and it will be more aligned across the project. (Another reason a deterministic workflow such as uv init is best, because it will always be the same).
2. build with clear instructions (agentic skills, superpowers, etc.) on how the tools are used (basically providing CLIs and API documentation). This is the bulk of the work anyway. That’s the data architecture, the brainstorming with fellow humans before you build something, instead of missing a key insight in the beginning and then letting the agent run down the wrong path. Also, be realistic: prompting “be correct” or “use state-of-the-art” won’t make it more correct or more state-of-the-art than the model was trained on. So if it’s a rather new architecture, it’s a must that you provide these links and hints.
3. set up the project in a modular fashion, so the agents cannot break the whole project if they make a small change, so you don’t end up in a scenario like dependency hell with everything dependent on each other.
4. use a declarative approach, with descriptive configuration that says the what and not the how, so that you can collaborate on these configs with the agents, version them, and easily revert or change something, as well as decouple the implementation logic from the actual business logic.

With these steps, you can get the best out of the agents of today. I’d say the model matters less, but the structure does, and as Mario says, so does the workflow approach. For example, extensively plan (the process before writing a single line) and correct the model before any implementation that could lead down the wrong path is written.

Also, don’t overthink it. But this is only the workflow and learning the soft skills and discipline of working with agents. How does that look in a real-world project?

[!note] The key is to get use out of AI, not to get more work.
E.g., most developers used to think about the problem. Today, most drown in PRs. When the AI tooling gets better, AI can provide more quality code that is correct, that needs less review or fewer iterations, which means fewer PRs and less work for the developers to go through.

The Correctness Layer for Data Engineers

A key insight is that AI agents should support the “human in the loop” for correctness, or a correctness layer. And rather than making more work to verify more code, we should be confident in the process and know that the code it produces is verified and ultimately correct.

But how do we get more “correct” work and a layer in which we can verify it? The biggest argument is a deterministic-validation architecture in full. E.g., Altimate Code splits the agent into a probabilistic layer on top and a deterministic Rust/TS layer underneath that does the actual SQL ops such as parsing, validating, and equivalence checks, so that the agent itself never has to be trusted on those questions.

Altimate Code, for example, is built on a probabilistic agent, deterministic harness, and deterministic core. The probabilistic agent with the LLM does the creative work of reading intent, picking a strategy, drafting SQL, summarizing results, and recovering when something goes wrong.

Below the boundary sits the deterministic harness, a TypeScript layer that intercepts every tool call: a dispatcher checks hasNativeHandler before the call runs, and routes it either to a native, deterministic handler or back to the model. Those handlers don’t reimplement logic themselves, they call into the deterministic core, a Rust engine (altimate-core) that exposes SQL operations as pure functions over ASTs and schemas, wired in via napi-rs bindings. Parsing, validating, transpiling, checking query equivalence, diffing schemas, extracting column lineage, diffing rows across warehouses — all of it runs sub-millisecond, and all of it returns the same answer on the same input, every time.

Like a compiler, the agent never decides whether two queries are equivalent or a column exists upstream. Instead, it calls a function that proves it against the parsed AST and the schema, the same way a type-checker proves a program compiles rather than guessing.

That’s the distinction that makes the output easier to review, as factual checks have been run and the output is either correct, or there’s a bug that it can fix directly. The rest a human can re-verify. On the dilemma of having stopped to hand-write code and approving it faster than humanly possible to check, you can also read more at You Are the Trust Layer.

[!note] There’s another factor, being wrong
Bare agent use might be cheap, but only until they’re wrong, and then the cost is unbounded.

Improvements for Better Usage of Tokens

Altimate, or data engineering agents that have deterministic functions and integrated understanding of how to work, can help you save tokens and be token lean (the opposite of [[tokenmaxxing]], which is popular on Twitter/X, using as many tokens as possible and having an agent running at all times). Because in large enterprises, token costs are a real budget point.

To slow down the tokens, an easy trick is to instruct the model to use fewer tokens and words itself - caveman is a good example of that, but you can also add a singular prompt to your CLAUDE.md, Codex, or model of choice in combination with Altimate Code.

There’s a second, less obvious cost: the token itself isn’t a stable unit. When Anthropic shipped Opus 4.7, the same prompt that cost X tokens on 4.6 started costing roughly 1.4X (same input, same answer, more tokens, same price per token).

Altimate on The Great Token Heist of ‘26 makes the case that “cost-per-token is the wrong number to optimize”, since the meter itself can move with a vendor’s next model update, and what we should track instead is cost-per-task. I fully agree, and this is where deterministic function calls work around that volatility by not using a model/tokens for every task, making it less expensive.

Typical Use Cases

In this chapter we go through typical AI agent use cases for data engineering.

There are many of them. You can use them to educate yourself or your team, build production data pipelines, build data apps, and visualize your data in new innovative ways (usually HTML web pages with React and other JavaScript frameworks). But in general, the use cases fit into these approaches:

1. Start a new project from scratch example: Building a data landscape with more open source.
2. Extending an existing project or data warehouse: Adding new data pipelines.
3. Maintaining current setup: Update and verify it still works when changes come in.
4. Migration: Migrate from one database or tooling to the next.
5. Finding the Blind Spots: Two similar-sounding IDs might be wrongly used for a join, or missing data in a column that got missed in a nightly load, or anything in between. If agents can do these checks, that would be super beneficial. With more access to CLI, Model Context Layer, and deterministic tooling, these things are truly possible.

Below we go through extending and changing an existing warehouse with a change of column, and using Altimate Code to give us a Blast-radius assessment.

Showcases: Blast-Radius Example

A Blast-radius refers to the potential extent of damage. For example, before you knock down a wall in your house, you want to know if there’s plumbing behind it, electrical wiring within it, or if it’s holding up the floor above.

The same is true for a data warehouse or a data project with lots of ETL. For example, if a data engineer cleans up the table fct_orders by joining orders to order_items and summing order_total. It compiles, the dbt tests pass, nothing errors. But the join changes the grain, so any order with several line items now gets counted once per item, and revenue quietly inflates.

It’s best to know, before you rename a column or add a new join, the downstream (data that comes after the current task) dependencies to the dashboard — that’s what the blast-radius report does.

With Altimate Code we can achieve this. Before any change goes through, it maps out the full impact automatically and produces a detailed blast-radius report with what will break, what’s safe, what needs someone to sign off, and also performs the changes. Here is what this looks like:

Rename and Change Columns and Logic

As an example, in this prepared ecommerce repo with different DWH layers such as staging -> intermediate -> marts, I prompted this request to change unit from cent to dollars:

It recognized the dbt name and invoked dbt-analyze automatically:

It gave me a full Blast-radius report and the impact my changes would have on the project:

Including semantics only, to point out what’s safe and what’s not:

With a fixed order to address breaking changes, semantics and docs, and intentionally untouched:

Notice, I hadn’t said anything about blast analysis or using dbt-analyze — it did it on its own, ran dbt, and analyzed it deterministically.

This shows how Altimate Code looks behind the walls of data engineering, just like blast radius analysis.

If you want to see another example and a full blog post on Blast Radius, check out Blast Radius Analysis Using Altimate Code, and what Altimate Code did as in the video. Or Altimate provides many more examples and Showcase on their website such as Migrate SQL Server to Snowflake with dbt or showing how to resolve An Upstream Schema Changed.

[!example] Connect a model to Altimate
Make sure to connect to a model with /connect and choose an existing subscription with API credits, or any other subscription. I used opencode zen for my example, which includes e.g. Opus 4.8.

Correctness Over Confidence

I hope you got a better understanding of why AI agents can be genuinely useful, especially when provided with the right tools and applied with the right discipline.

You’ve also seen how deterministic tooling, purpose-built for data engineering and analytics problems, gets you both better correctness and better token economics than general-purpose agents alone.

Coming back to where we started: not every task needs a level-three agent. A quick chat-phase agent is fine for exploring a dataset or drafting a query you’ll review yourself. But the moment that output touches production or serious work, a dashboard, a nightly job, a number someone makes a decision on, you want the deterministic core underneath it, not just a model that sounds confident.

That’s the gap Altimate Code is built to close. It runs on deterministic functions purpose-built for DE workloads, it’s open-source via the OpenCode TUI, and for teams wanting more, there’s Altimate Studio — a paid, multi-agent platform with extras like warehouse cost optimization, dbt development acceleration, and migration tooling.

–

Check out Altimate Code, it’s free and open-source. Give them a star if you like them, and find more information on their docs and new website.

Full article published at Altimate.ai - written as part of my services

So what is important when taking notes? And how you might take notes too? First, the book [[Smart Note Taking]] by Sönke Ahrens helped me tremendously to understand the basics, that was after I was taking notes already for 10 years, but the principles of it being fine to delete notes, to go from literature notes to evergreen (though I changed that a bit for my liking), were two things that helped me a lot. To see the notes as a system of thoughts, more than just notes. From dumping all my thoughts into a blank file in my text editor of choice to sorting them, and getting clarity from it. And that writing is not the outcome of thinking, it is the medium in which thought occurs. This has grown since then, even more.

But if we think about it, we can also make it a hugely scientific and philosophical, but on the other hand, it’s just ideas we have in our heads to:

1. Write down in a way we don’t forget, and more importantly
2. find them again in a year or two.
3. Also, do not write only a couple of words, as later you might not know exactly what you meant, but also don’t write too many details, or worse, copy a full article.

The key is to pause for a second, and think about it, and then write it down in a sentence or two. I usually use listings, so I know it’s kind of a brainstorming, if I want to go deeper, I can just indefinitely indent and go up again. So I can, if I want, write a full outline, add new thoughts from the idea I just heard, link it to other ideas and notes in my vault.

But number two is harder than it looks to write it in a way to find it again easily. That took me a long time to optimize, and only changing of system from OneNote to Obsidian, and more so from file organization to connected notes, or also called [[Zettelkasten]]. This means, during the writing, I didn’t need so much thought where to put the note, is it in business, or personal, should I create a new folder, or do I need to search all my notes if there’s already a similar folder, to just open a new note, doesn’t matter where it is, directly type, link it to at least one or two other notes (so I find it later if I forget the file name based on backlinks) and give it a name that I might search for in a year. That is crucial, as I almost exclusively search all my notes via file name, the name is key.

Luckily [[Obsidian]] helps us with duplicated names, because if you open double brackets anywhere with [ [ and start typing, you find all notes with that name, so you don’t accidentally create a note twice. And for the finding part, if I know I have written once about it, but I can’t find the name, I search for the content with search or other mechanisms in Obsidian. But once I’ve found it, I directly rename the note to something I was searching initially, as this means to me, the name was not good enough yet, or add a keyword. So next time I can find it immediately.

The Process of Note Taking

There are different stages of note taking, the first part of capturing your note that we just described. But then there’s the refinement and the publishing phase. Tiago Forte, for example, calls it CODE (Capture, Organize, Distill, Express).

While you can add a lot of work for yourself with telling yourself you need to refine your notes, take time every week to process the latest, I do none of that (scheduled or planned). I believe the note taking should help me and not add more work to me. Sure, I refine them, and a lot actually, but it’s based on usage.

I look for signs. For example, I come back to a note over and over again, or I looked up a note two three times recently to understand a concept better, or I linked to a note many times. All of these are signs that this note needs more attention, potentially even splitting it, really refine, and then potentially publish on my public second brain (I can just add #publish, and it lands on my website), or write a full blown blog post on my blog website.

The Key of Format

One thing that unlocked so much more of the sharing part, and joy of using my notes, was the switch from the lock-in format in OneNote to an open [[Plain Text Files|Plaintext Files]] file [[Markdown]]. These files are just text files locally on my machine, meaning I can open them with my [[neovim|Text Editor]] or [[IDE]], or do a search and replace on all my files. Back them up easily, and also notice they almost use no space, as just text files. And the best part, I completely own them, so all my ideas and work I put into them, I will own forever as long as there will be computers. They will not die with the tool such as OneNote, Apple Notes or Notion, once they are gone, your notes are gone too, or if they deactivate your account, byebye all your notes.

Blogging

But the not foreseen effect, was that when I migrated my blog from WordPress ([[Rich Text]] interface) to a [[Static Site Generators (SSG)]] with [[GoHugo]], that is just Markdown. Now instead of needing to reformat, to upload somewhere in an online form, and constantly need to sync them between each other, say I updated my notes or fixed some grammar. No, now it’s essentially the same note. Obviously, the blog post needs a title, a hero image, maybe a description and some more attributes, the plain note doesn’t need.

But once I noticed this, I upgraded the workflow quite a bit. And I added my [[Quartz]] based second brain to my website (see ssp.sh/brain), and there I had the goal to do zero edits, and just hit a button and publish. And to be honest, that’s what I do for more than 6 years or so.

I add #publish to my note, and run a script (it ended up not being a button :), and it’s on my website under new recent note. Including all the backlinks I added via Obsidian and formatting, links title and everything. I even added an [[Dynamic Feature Images|automatic hero image generator based on each note]], and updated a bunch of more advanced things with the frontmatter that Obsidian later added too, same as my GoHugo had already, meaning the format aligned even more, and is even more the same!

What I’ve Learned

So what I’ve learned? It’s really important to remove friction, to make it so that you can use and enjoy it. Maybe for some it’s needed the friction of opening a web form and typing in there, same as some like writing the email newsletter in an email client, to feel the pressure of the readers more, than just in a text editor - which I also implemented in [[neomd|my email client]] and newsletter in newsletter-rss, I can either run a script and start an email and upload it to my email newsletter, or write it to newsletter-book@ssp.sh and my email client will create a new campaign in my listmonk and send it out 30min later.

But that is what I was saying, the pressure of writing in the email or WordPress editor directly can be a help, but if you want to [[Learn in Public]] and share what you learned more often, I think removing friction is key. Also to overshare some times, when the note is still very rough. Because people might give you feedback, that helps your thinking or gives you an angle that you never had before.

TL;DR

1. Take smart notes
2. Use connected links instead of folders to break out of the 1 dimensional world of folders, especially if you write or want to learn. (it takes time though to adapt)
3. Choose an open format, I highly recommend Markdown. Also read [[File Over App]] as Kepano said, the app might not be here in 5 years, but your notes will.
4. Publish notes to learn, make the process as smooth as possible, no formatting or copying/duplicating of your notes needed.
5. Have fun, and enjoy the process. Especially with AI agents, choose a framework like GoHugo or Astro that builds on Markdown and plaintext files, and get the agent to help you with making the site pretty, adding features to the website that you like and make it truly your site. Not a boring site that everyone else has, not for others, but for you.

This was a little bit of a ramble, but it’s really a love letter to my 17 years of note taking. It [[My Note-Taking Path|evolved quite a bit]] sine I started, but stabilized now for quite a while with Markdown and open format with GoHugo.

If you are curious how the notes look, just go to my second brain and check out my public notes and how all my notes are public and connected with an interactive graph and backlinks. You can literally dive into my brain, see what I am thinking about, see how all the notes connect, hopefulyl learn something new for yourself. If you do learn something new, please reach out and write me an email and tell me. I love connecting to like-minded people.

Latest addition to my website here, I interconnected not only my second brain with the graph, but my blog and my book I’m writing at dedp.online, on each chapter, blog or second brain note there is an interactive graph, and you can see links to the second brain and vice versa. Check this note as example, or see [[Future of Blogging]] where I elaborate a bit more on that topic, and included backlinks.

How did you evolve your blogging and note-taking over the years? Any other learnings? I also love the Obsidian Workflow with templates I use everyday and the tools to write in with Obsidian, or check an older article on the same topic Personal Knowledge Management Workflow for a Deeper Life. Check out my other blogs if that interests you. I also share templates in Obsidian, or to clip online articles with [[Obsidian Webclipper]] and its templates.

The whole second brain code is public at my second-brain-public and how I automated the copying of my obsidian notes to my second brain, so I don’t need to duplicate them, is described here [[Public Second Brain with Quartz]].

In Part 1 I argued that the open data platform makes orchestration data and its flow simpler, providing integration into data assets and transformation, ETL, BI and even Kubernetes all while being open, making it a fully composable data stack. This piece continues with what it takes to operate orchestration, and looks at one of the hardest parts of data engineering “DevOps”, and how to manage different environments, business code vs infrastructure code, scalability and how orchestration.

I’d argue that DevOps is what data engineering was a decade ago, when everyone was talking about data science, but the actual work was 80% data engineering. With AI and generative AI, where even more data gets used for model training, or generated by AI, data engineering gets even more critical and larger, therefore also DevOps, which over a certain size gets the biggest task: deploying and upgrading to the latest error-free version, and keeping it stable and scalable.

Deployment, how to Start: DevOps vs. GitOps:

Before you start deploying or operationalizing anything, you will get into [[DevOps]] and more so, [[GitOps]]. That’s why we start with that.

Because once you start with deploying your own infrastructure, you need a way to automate changes across your systems, or even your development server. You can’t manually deploy on different systems. If you only have dev and production, it might work at the start, but there are so many dependencies, stateful NFS mounts where important configs or DWH data is stored, or dependencies between the installed tools from the BI tools to data integration and orchestration.

GitOps is taking that deployment git repo and including infra code such as Kubernetes deployments (also called Infrastructure as Code (IaC)), and making them automatically deployable on git commit pushes to the git repo with post-trigger actions. With the goal of operationalizing data orchestration, we need to look at GitOps and DevOps best practices. GitOps is everywhere these days, with agents working best with Markdown or local YAML or code that is checked in to a git repo.

Orchestration is no different here, we want to have a repo for the code of data pipelines as well as infrastructure and ENVs for each environment (prod, test, dev). We need to separate infra from business-heavy ETL in the data pipeline. We want to standardize the deployment patterns (e.g. base and overlay is how Kustomize is doing it).

How to Operationalize Data Orchestration?

So now that we have GitOps ready and understand it, let’s revise the challenges and then best practices of operationalizing a data orchestration tool. To make it more concrete, I use [[Dagster]] and its features whenever suited.

The Challenges of Deploying Data Orchestration

One important part is how to deploy a system that will be at the heart of your data stack and responsible for the refreshing of your data assets.

The problems are multi-fold. Do you want to auto-scale with Kubernetes, Terraform, meaning you rent machines and scale up and down based on tasks in your data pipelines, spawning up multiple spark jobs, multiple DuckDB processes in parallel as part of spawned containers? Or do you want to pay for it and just connect data pipelines with Databricks Spark Cluster, and hook up Dagster+ that handles it all.

Compare it to the Titanic iceberg collision where too many holds overflowed and then it sunk. Alex Noonan says it’s the same with data at scale, you get too many duplicated data pipelines, tools, inconsistent data contracts, unclear ownership and ungoverned sprawl.

[!note] A handy feature built in: cost observability for cloud
Dagster’s ability to track compute costs per asset, per partition — this feeds directly into your “open data platform” narrative. FinOps for data pipelines.

Best Practices for Running Data Orchestration: Organization vs. Deployment

The problems at scale can get complex and run into a lot of work, quite quickly. Therefore we need a clear setup that works for all our deployments. Here we look at separating the organization-side and the deployment itself.

It all starts once you decide to run your own deployments and machines. Maybe you have a platform team or cloud server that you use with Kubernetes. That’s when we need to separate the state vs stateless, and also the code of data pipelines is a key point here.

1. Organize DevOps means to focus on structure and organization. We need the right organization for structure and repositories. For example, I once created a Kubernetes GitOps deployment Blueprint repo, that includes such structural questions such as GitOps workflow patterns, infrastructure configs, tenant isolation, database migrations, and observability templates for production deployments with Flux.

The organization usually happens in the form of a git repository, and you have one for environment and deployment for infrastructure as code. You set up some sort of GitOps with Flux, ArgoCD, Terraform, or GitHub Actions or similar to automate deployment with a single git commit && git push. All of the environments are typically handled with ENV variables.

2. is Deployment, this is where we handle state and stateless. The deployment repo is usually mostly stateless. It has state for persistent storage locations for configs that Dagster, or BI tools or others need, but these are of a technical nature and usually can be recreated as there is no historic data stored as in a data warehouse where we ingest data from source OLTP databases for example, and version the changes. This way of not breaking state is much harder.

We need to make sure that the data assets produced by the data orchestrator stay intact, and don’t get purged with a new deployment. Other things to take care of are technical resources such as the Spark integration as a resource Python file, which is stateless and can just be redeployed when we update the code, so we have the latest version prepared for the next run.

There’s no complexity, and with proper versioning, we can also easily roll back by just setting the older version in our deployment repo, commit and push. GitOps will then make sure to roll back to a working version. And when the data itself needs repair, partitioned backfills let you re-materialize a specific time range across the whole asset graph, with dependencies resolved automatically.

Code Locations: the Key Piece to Connect State and Stateless

The key piece, again, is connecting state with no state, and Dagster offers an elegant solution here that I suggest for any data work: code locations or also thought of data engineering workspaces. The code locations connect the state and stateless in a meaningful way. Code locations are both stateless (libraries used, technical implementations) but also stateful once the business logic has been run. The outputs are data assets used by downstream processes, and we can’t just delete and recreate them easily. It might also be that some of the assets take hours to create, so just deleting them and then rerunning the full data warehouse for a day isn’t desired.

Code locations are key to a successful orchestration deployment story where we differentiate business from technical implementation logic. Many start deploying Airflow or other data orchestrators and pretty quickly land at the questions: “how do we update data pipelines?”, “do we give people access to the deployment repository?”, “where are all the envs for each environment?”, “how can business users quickly test their Python script or notebook?”, “what about access permissions to the data assets, are they allowed to see all tables, the full S3 bucket?”.

With workspaces you can containerize your data pipeline logic, with its required Python libraries, into a docker image or similar, and test it and deploy independently from the deployment code that is usually only accessible to the platform team or data engineers if they do platform engineering. This leads to a natural separation between data wranglers and data platform people.

Here’s an overview of how centralized teams can structure their code locations differently - from centralized to embedded and to hybrid, with dedicated workspaces for different teams such as data engineering, data scientists, finance or marketing:

The differences are that centralized data team structures can give you consistency across teams, but are bottlenecked and slower when you want to change and iterate. Embedded means that teams have a link to each team, increasing speed and domain depth, but can lead to drift and fewer standards if not careful.

The different setup of teams is also something we have discussed since the dawn of data work. It started with a centralized data warehouse to a more shared Data Mesh and other approaches. With code locations, you can have a centralized data platform orchestrator while still having your domain ETL and data pipelines in dedicated team code repositories, using a hybrid with the best of both worlds.

The code locations also fit into the right abstraction layer as we talked about in the Dagster Almanack, meaning that code locations separate concerns for data teams, and with Dagster resources separate business from technical implementation, that can be used across different code locations (e.g. versioned PyPI packages), making the setup more flexible and adaptable to how data teams work.

Microservices that Work with a Central Open Data Orchestrator

The right abstractions with code locations let you use data pipelines as an advanced microservice with reusable technical code while using sole-purpose domain logic in each location, that integrate into the platform easily with Dagster abstractions and applied functional data engineering best practices. With something like Dagster Pipes, you can invoke external code — Spark jobs, notebooks, R or Rust processes — while logs and metadata still stream back into the asset graph.

If you need a better integration for less technically savvy people, you can use the in-built DSL. DSL stands for Domain Specific Language, which lets you define your pipelines and transformations with a simple YAML interface. Dagster Components takes this further: a toolkit for packaging integrations and common patterns behind a YAML interface, so pipeline instances can be defined with little to no Python.

[!note] General Learning from GitOps
More learnings from using GitOps and deploying orchestration and general data workloads with Kubernetes, here are 8 more points:

1. Separation of concerns is crucial - keeping infrastructure, platform services, and business logic (pipelines) in distinct sections makes maintenance and collaboration easier.
2. Standardized deployment patterns like the base/overlay structure with Kustomize allow for reusable configurations across environments with minimal environment-specific overrides.
3. Versioned artifacts with timestamps (e.g., finance-pipeline-20250512123045.tar.gz) create a reliable release process that enables rollbacks and audit trails.
4. Database migration automation tools, such as Liquibase, can handle schema changes programmatically across environments, thereby reducing manual errors.
5. Test early, test often - validate data pipelines, infrastructure code, and database migrations separately before integration testing in an isolated environment.
6. Workspaces separation from infrastructure code enables domain experts, such as data scientists and analysts, to focus on their core competencies while maintaining deployment standards.
7. CI visibility through lineage diagrams and documented processes enables teams to understand the deployment flow and troubleshoot issues more efficiently.
8. GitOps, as the single source of truth, means changes occur through Git commits, creating an automatic audit trail and enabling pull request reviews.

This overall blueprint, with best practices, helps you specifically with data engineering reference architecture for Kubernetes-based data platforms using GitOps workflow patterns. It includes infrastructure configs, tenant isolation, database migrations, and observability templates for production deployments with linked CI/CD implementation with GitOps infrastructure example.

How to Deploy Dagster: Kubernetes, Helm, Docker

If we go one step deeper, we see different ways of deployment pretty quickly. We will keep this chapter short, as it would fill a full book, but generally you need to decide if you want to run scalable on something like Kubernetes, where pods and compute are automatically scaled up/down, or locally to test, with docker.

Or if you just want to get started without headache and lot of development time, this is where paying for something like Dagster+ service makes sense.

The biggest question for data pipelines is where do you run the compute, do you scale up a Spark cluster, do you run Python or SQL engines as part of Kubernetes pods, or do you use any other way. Because that’s where you will spend most money, as data pipelines are only expensive as the process they run, e.g. crunching large data sets with compute. Partitioned, incremental materialization is the other half of the cost equation. Processing only yesterday’s partition instead of a full refresh is where the orchestrator saves compute money. The same scaling applies to partitioned asset checks, so quality validation only runs on the partition that changed.

You can think about it more high-level, so segregate storage from compute with a Lakehouse or with tables and data stored in Open Table Formats (Iceberg, Delta and Hudi), that saves you money by not using expensive SSDs, but is slower and costs more on the analytics side when you query and join these data.

Here’s a list that helps you get started to operationalize Dagster as the data orchestrator with different deployment options:

1. Helm Charts (recommended for production on Kubernetes): Official chart at helm/dagster: helm repo add dagster https://dagster.io/helm. Two charts: dagster (webserver + daemon infra) and dagster-user-deployments (your code locations), so pipeline code deploys independently of platform upgrades. Guide: Deploying to Kubernetes.
2. Docker Compose (local or single VM, e.g. EC2): one container each for webserver, daemon, and per code location. Example: examples/deploy_docker, guide: Docker deployment. For pure local dev, just dagster dev.
3. Kubernetes customization: there is no official operator — the Helm chart is the K8s story. Runs launch as K8s Jobs via K8sRunLauncher, with autoscaling configured and tunable per asset/job with dagster-k8s/config tags. See Customizing your deployment.
4. AWS ECS / VM (if no K8s): Terraform/ECS pattern, example at examples/deploy_ecs, guide: Deploying to AWS.
5. Hosted: Dagster+: Serverless (they run everything) or Hybrid (control plane hosted, an agent runs compute in your K8s/ECS, deployed via the dagster-cloud-agent Helm chart).

Guardrails and Pitfalls

Beyond deploying and operationalizing the orchestrator itself, there are related topics that cut across data orchestration, some of which we touched on in the open data platform architecture image in Part 1, where you see related tasks we need to deploy when wanting a full open data platform. The most important are testing and governance.

Governance & Testing

A deployment is only as trustworthy as the data it produces. With integrated data quality checks, we enforce assertions on every run instead of testing data after the fact, something a BI tool can’t do, and the source can’t either, because the business transformation hasn’t happened yet. The data pipeline is exactly where the business rules live, so it’s where domain experts can enforce checks and data types they know must hold.

With asset checks, this even extends to assets external to your pipelines. Yes, writing checks is extra work, but it makes the whole system more stable and less complex: numbers are known to be correct, and you avoid the downstream ad-hoc fixes that persist as duplicated data marts or gold-layer (if we use Medallion Layers).

Testing data isn’t comparable to testing software, as data, tools, and volumes change between dev, test, and prod, and upstream from source data sources that we have no control over.

This is again where abstractions make life easier, e.g. Dagster separates business logic from technical implementation in resources. You can swap a mocked or local Postgres resource in dev for Snowflake in prod without touching pipeline code. Combined with the environment separation from our GitOps setup above, functional tests and balancing tests (does the count in the source match the warehouse?) can run in Continuous integration (CI) before a single git push ever reaches production.

Governance goes beyond quality checks: asset-level access controls, audit trails via metadata, column-level lineage for GDPR, and PII redaction in compute logs are the features that make a deployment enterprise-ready.

Obviously, testing and governance are their own domain and field, so that’s worth another article, but I didn’t want to leave it out, as it needs to be part of the strategy and best practices. For example, simply defining what your prefixes for table names in stage, or core, are, or how to name last_updated and created_date across the warehouse and deployment of your data platform or orchestration is key.

Provisioning and Multi-tenancy

Data orchestration is what Dagster is made for, but provisioning is something that fits well in the scope of using Dagster and deployment. As a pipeline rarely runs alone, it also needs an object store, a data catalog, maybe a Spark cluster. Provisioning these accompanying services is itself orchestration, and platform teams increasingly use Dagster for exactly that. It is also an advanced, observable cron for infrastructure. In several organizations, this started with data pipeline deployment, and spread across the company from there.

So without provisioning the needed compute engines or related tooling for your data pipeline, operationalizing the data orchestrator and the full open data platform is not complete. This is also where in larger companies this task usually falls to two different teams, data practitioners (teams of data engineers, analysts, or power users) and the data platform team. The two teams are different, but they have the same need to orchestrate their many tasks and jobs:

For enterprises, this pairs naturally with multi-tenancy and branch deployments: isolated environments that mirror production, so teams can review data pipeline changes like code PRs, and the GitOps workflow from the deployment chapter is applied to the data itself. A concrete example: Lakeflow Jobs can’t express dependencies across Databricks workspace boundaries, but bringing every workspace into a single asset graph can.

Orchestration as the Operating System of Your Data Platform

Closing the initial Dagster Almanack with its guidance, insights and tips for data platform engineers, as with this operationalizing data orchestration guide on DevOps, Infra, and Code Locations in the shared task of deploying a full open data platform.

We have seen how to operationalize data orchestration, and what features Dagster can bring to the table. The best practices are to organize and deploy around git repositories and with separation of concerns in mind. When you think about deployment, always ask yourself where your storage of data assets will be, and how to compute them. This will define the best strategy for finding the best platform for deployment.

The open data platform deployment also helps escape the modern data stack trap. Dagster helps you integrate different tools in line with deployment strategies while separating workspaces for different teams and domain experts in an elegant way that works well with GitOps strategies. Exchanging compute engines and using the best tools for the task is just one config change away, and doesn’t need to change the logic of the data pipeline itself.

With newer features such as state-backend components, Dagster uses these for integrations whose definitions depend on external metadata. Instead of re-querying external systems every time a code location loads, Dagster can fetch that state at controlled times, persist it, and then build definitions from the cached result.

It makes it a composable toolkit for data engineers out of the box and a single pane of glass for the data platform, incrementally adopted across teams and stakeholders. Tools are integrated into a single data platform, with a single deployment strategy, where multiple teams work together.

In the future with more shifting left, DataOps that combines this all under one domain, and with AI agents that govern and help create data pipelines, this foundational data platform is more critical than ever, integrating and scheduling all tasks in a structured and reliable way.

Full article published at Dagster.io - written as part of my services

Creator of Pandas, probably the most widely used data analysis library for Python, Wes has shaped the era of data and is co-creator of Apache Arrow. He also created Ibis to address these issues with a different approach to Python dataframe libraries, by decoupling the dataframe API from the backend implementation.

The article is structured in four parts: (1) how to trust the outcome, (2) knowing what not to build, factoring in cost-per-token among others, (3) accountability of agents and the code they generate, and (4) philosophizing about the future of agentic engineering.

Introducing the Guest: #3 Wes McKinney

Besides creating the most popular dataframe libraries used by most data people, Wes McKinney now focuses full time on agentic engineering with his newly founded company Kenn Software, which focuses on the promise of building a new stack of development and knowledge systems for the agentic era. He’s also doing AI and Python at Posit, where they work on a data science IDE. He’s a part-time investor in various startups.

Wes has been running Claude Code, Codex, and Gemini CLI for months. Thousands of sessions, hundreds of thousands of messages. He has released multiple tools that help the agentic work (more on this later), and he is at the forefront of what’s going on with his recent blog posts about “Why he uses programming languages built for agents, not humans” and Mythical Agent Month, with his recent insights into how to work with agents. Find all his takes at Wes McKinney.com.

I had the pleasure of asking Wes more about these topics, and we’ll go into more details, plus many other things. Let’s get started.

How to Trust the Outcome?

We started the interview with a critical question that stands above all others in the current AI landscape, and I asked him: “Can we trust the outcome?”. What if we need something important, other than a hobby project? What if the data must be correct (hospitals, banks)?

Similar to what Mark Freeman told us in our last interview about using spec-driven development with spec-kit, Wes uses a similar approach, but with an agentic skill framework called superpowers (currently 216k stars on GitHub). Compared to spec-kit, it specs out the requirements differently by (A) guiding you through the conversation, asking you the right questions to get to what you want to build, and (B) once you fire it off, it spawns a sub-agent that keeps the implementing agent on track. Wes said, “Superpowers looks for drift”, and course-corrects if the implementing agents drift off to non-relevant, or not even specified, tasks.

Wes spends a lot of time in this specification phase, sometimes hours, very detail-oriented and engaged. Even before he starts speccing, he has subconsciously worked over the topic and idea for a long while. He will not start implementing something when he doesn’t know super clearly how it fits together. The insights, the architecture, come from him. But the interview style by superpowers helps him clarify his thinking.

He doesn’t only give his feedback to the questions, but sometimes also fires up multiple agents and integrates their feedback. Codex models especially seem to work well for design questions.

He puts a lot of importance on the spec being:

1. Spec conformant: Meaning the agents act in accordance with your specific set of rules, standards, or specifications.
2. Code correctness and quality: This is where Wes uses e.g. Roborev, his own created AI-reviewer.

Correctness is crucial, which led to creating Roborev. Wes developed many tools that help him work agentically, and we’ll hear about many more later. Roborev, for example, is a code reviewer that can be initialized with a hook on a git repository, and from that moment on, every commit will be auto-reviewed by Codex (the default, but you can choose others too).

I use Roborev myself, and this is what the interactive TUI looks like - showing the most recently fired hooks with their running status, but most importantly, whether the review passed (P) or failed (F):

If it failed, you can open the review and see detailed findings categorized into severity low, medium and high:

The convenient workflow is that you copy the review with y and feed it back to your running agent to let it fix things directly. The current agent that created the change works best, as it already has all the context, compared to starting a new one that needs to load context and what has been done.

Roborev also helps to review a smaller part at a time. Wes also says it will never catch all the errors, but LLMs are very good at pattern matching, which is what error finding is, and they find many that might be missed. On top, he adds reviewers with different roles, e.g. giving agents roles such as focusing on security, CI, software development, or performance, which gives much more accurate feedback than a general reviewer.

After having gone through the spec intensively, having made sure that drift happens as little as possible, and having auto-reviewed each commit by Roborev, what is left for him to review is much less now, and of high quality. He then reviews the code and checks that it looks and does what he expects or envisioned.

Wes has a very clear problem or idea that he then solves meticulously. However, at the same time, he runs agents in parallel and works on many projects concurrently, context-switching between them¹.

[!note] Rigorous process in place needed: Changing models

The rigorous process he does is also needed because the models are constantly changing and are very unpredictable. It is hard to have a consistent outcome if you do not have reviewers and processes in place. And Wes says these AI reviewers are much better than just static analysis.

How to Maintain Agentic, or General Projects over Time?

The second question was about maintaining projects and how Wes handles maintenance, as creating projects is usually the easy part, but maintaining them for years to come is difficult. And how does he see that in combination with AI? Will that be outsourced to AI?

First of all, Wes uses his own projects and tools. That’s the reason they exist, and it helps him find bugs. This is why he fixes errors or bugs when he runs into them. Besides Roborev, which helps tremendously to review and have fewer errors while developing, he uses Middleman to keep an eye on his agents and projects. It’s another tool he built that gives him a local-first GitHub dashboard and triages what to maintain or fix from other users.

He automated repetitive work such as releasing with a full release script so he can release fast and fix bugs fast. The Changelog on GitHub is fully streamlined, too. He is also careful about what comes into the main branch, only changes he has verified and assessed as “pass”.

To illustrate what Wes is maintaining, here are some of the projects Wes built recently, some of which he might not have built without AI:

• roborev: Continuous code review for AI coding agents. Runs in the background and surfaces issues per commit before they compound.
• middleman: Local-first GitHub dashboard for maintainers to triage, review, and merge PRs and issues across repos.
• agentsview: Local coding agent session viewer for Claude, Codex, and Gemini with analytics and full-text search.
• msgvault: Archive a lifetime of email and chat locally, with full Gmail backup, DuckDB analytics, a TUI, and an MCP server for AI queries.
• moneyflow: Personal finance data interface for power users, supporting backends like Monarch Money and YNAB.
• Spicy Takes: LLM-analyzed blog posts from 20+ prolific tech writers, each with a TL;DR, key quotes, and a spiciness rating.
• VibePulse: Simple macOS menubar app to monitor Claude Code and Codex token consumption.
• kata: Local-first issue tracker for AI-assisted software work, with an agent-friendly CLI and human-facing TUI.

[!note] Earlier Tools and Frameworks Wes Has Built

• Positron: A next-generation data science IDE built on VS Code, supporting Python and R.
• pandas: The most widely used data analysis library in Python.
• Apache Arrow: Language-independent columnar memory format for analytics.
• Ibis: Portable Python dataframe API that works across any backend.

Building for Maintainability: Modular?

I asked him if he builds for better maintainability, e.g. builds in a modular way so the AI agents can easily fix something or create a feature in a dedicated area without breaking the full program.

He didn’t answer the modularity part directly, but Wes implements and uses tests extensively. If something needs to exist, he writes a test for it. But even more, by investing in test infrastructure, regression tests help prevent bugs and protect existing features during rapid development.

He also mentions that bugs are created faster these days, but also fixed faster.

How to Decide what to Build? Saying No!

Given that AI can get addictive, and in a time when you can build almost anything, I asked Wes how he knows what to build, and when to say no to avoid building the “wrong things”.

He said that:

It’s not the ideas on their own, he’s thinking a lot about what he wants to build.

Again, it is in his subconscious. He thinks and asks himself all day: “How is it beneficial for agents? For humans? How can it be applied?”

If he can’t explain it, he will think more. For example, msgvault didn’t have a web interface, and he could have easily added one from the very beginning, but he didn’t have a clear picture. So he just postponed it until later, when he had a use case, a pain point, or a real need.

'Those are the constraints', Wes adds. 'Because if you don't, AI will bring in lots of crap'.

Superpowers also helps him with guardrails by keeping the AI on track. Besides, Wes has a perfectionist mindset, making him want to perfect the tool that works for him and improve the workflow.

When He Was Building without AI: Pandas

It was the same when he was building Pandas: he was building it for his use case when fiddling with Excel. Then there is taste.

Every prompt, every decision in the spec phase adds up to 100s or 1000s of small decisions, essentially manifesting one’s taste. That’s why the product comes out differently from two people, even though they use the same LLM models.

[!note] Find more at AI Council Talk about the Scope, Design and Taste in the Mythical Agent Month

Wes gave a very insightful talk at AI Council 2026 about this very topic, called the Mythical Agent month. He said what is left is “Scope, Design and Taste” with Conceptual Integrity (from the book by Fred Brooks).

Saying No is Our Last Defense

In his recent slides, he shares “When code is free, saying no is our last defense”:

Every new feature is cheap to create but expensive to maintain. Each one adds surface areas for bugs, confusion, and future agent mistakes.

[!tip] “Hell Yeah or No”: A similar term by Derek Sivers
Similar to Wes’s figuring out and saying no as our last defense, Derek Sivers said something similar before, where you say no to everything until you feel “Hell Yeah”. This Hell Yeah or No approach doesn’t seem to have changed much with AI. This doesn’t apply only to AI, but also to life and career, in my opinion.

Cost-per-Token at True Price Will Stop the Waste

A very current topic is how the growing cost-per-token factors into this decision of what to build. Or does it not? There’s even a term called token maxxing that encourages programmers to use more tokens, whether by the company or by peer pressure on X/Twitter.

Wes was at the top of the HN leaderboard at some point, currently on #4:

Wes’s current usage is ~$20,000/month at API rates, which he sees on another tool he built called AgentsView. He said that

He thinks that all his high-quality output through the shared tools or the work he does is higher than the invested money.

But on the economics side, he thinks that:

Subscriptions go away, and pay by usage, a good thing. AI slop and low-value projects go away. This helps pay the true cost of tokens, which isn’t the case for now, making the consumption (or even waste) of lots of tokens non-problematic.

Enterprise Token per Employee: Clarify Useful vs. Vanity AI Work

This was actually one reason why he built AgentsView: to have an overview of your own usage, a better “token intelligence”, but also at a larger company to measure each developer’s usage. It could be part of performance reviews, showing each user’s token spend vs the value generated.

You’d have to justify your tokens, the opposite of now, where developers at Meta or Amazon are expected to burn tokens without incentives. Right now, it’s the wild-wild-west (something previous interview guest Chris Riccomini also said).

Accountability of Agent-generated Code? Who is Responsible?

My next question was how do we make people accountable for things they didn’t create (vibe coded)? I gave the example of self-driving cars: who takes accountability if a Tesla hurts someone? (That’s one reason full self-driving is still not allowed in Europe, as it’s legally not settled who is accountable.)

Vibe Coding ≠ Vibe Coding: But Agentic Engineering

Wes made clear that what he does is not vibe coding, but agentic engineering. All the planning and architecting with superpowers and his newly created tools is not the same as vibe coding.

The term vibe coding to him means when you just one-prompt it, don’t look at the code, and ship it. Again, this is not what he does.

He says:

We can’t disengage from planning and writing specs. We can move much faster, but don’t vibe code. Vibe coding is very dangerous and irresponsible.

Like the Coinbase example, he finds letting non-technical employees push to production highly dangerous. We humans, with fundamental understanding and seniority, need to be more engaged in designing and testing, as coding is essentially “cheap” now.

He continues:

Automated code review certainly helps, but it isn’t a substitute for engineering experience.

Philosophize about the Future with Agentic Engineering

Wes is also an investor, a person who foresees the landscape well with his involvement in major data libraries. I asked him: “If you think about AI, where would you invest your money? What do you trust will have the most benefit or will work well with AI?”

Where do you see the future heading, or where does this end? Especially when we talk about data engineering?

Future of Data Engineering

He says that he is not involved too much in data engineering anymore, but that he is an investor in dlt, MotherDuck, and Bruin. But his main focus is on agentic work, somewhat on top of the “dbt legacy”².

But what he sees as currently the hot topic is Headless BI, custom dashboards, and building a semantic layer for better context for agents. Things like business rules and sending the “right” queries. Building new knowledge systems for companies. For example, through msgvault, which extracts value from years of emails and easily makes them searchable.

He saw people building personal CRMs on top of msgvault and their emails. That’s the current direction we are heading, he says.

How Do We Still Learn? By Learning by Osmosis

The challenge will be: how do we develop senior engineers without writing code anymore? Wes himself doesn’t write much code anymore, but reviews, guides, and adds taste. I asked him how someone can gain the work experience he has without the coding or going through the pain of coding, while avoiding the danger of not learning anything new, or getting overwhelmed with constant stimulation and potentially becoming addicted.

He says the hard labour goes away, which is where we usually learn. This is the way of learning by osmosis³, where we acquire knowledge while failing or naturally through exposure and immersion. He thinks the focus needs to be on design patterns and understanding architecture, to have the technical vocabulary to guide or understand the agents.

Next Interview

I hope you enjoyed this interview number 3 with Wes. Huge thanks to Wes for taking the time to speak with me and for sharing his experience with all of us. Follow him on Website, LinkedIn, X/Twitter, or on Bluesky, and follow along on his new company Kenn Software, or check out his agentic engineered tools he built at GitHub.

There is one more interview already lined up with none other than Maxime Beauchemin, so please share feedback, questions you might want to ask, or just your experience on how to work with AI in the data space. We’re all in this together, figuring it all out. The more we can learn from each other, what’s important, and maybe also what’s not, the better.

Full article published at MotherDuck.com - written as part of my services

So how do we make data agents reliable and accurate for database queries? Everyone talks about harnesses, evals, and context layers, but the real challenge is bringing them together with data engineering and the context you already have, such as database schemas, a semantic layer, metric definitions, plus the business knowledge that normally never reaches the agent.

That’s the question this blog tackles: how agents can work with the data stack and analytics, and how a context layer fits in. We also take an inside look at ktx, a new context layer that reads from the usual sources but also the less obvious ones (Markdown, Notion, etc.), driven by agentic workers.

The idea is to pull two kinds of knowledge into one reviewable place: the hard semantics (your warehouse schema, joins, and metric definitions as YAML and SQL) and the soft semantics (the business context living in docs, wikis, and Notion that agents usually never see). Both are committed to git and reviewed like code, so a human stays in the loop while agents get a warm start instead of a cold database connection. The payoff: more accurate answers with fewer (and cheaper) queries against the warehouse.

SOURCESWarehouseschema · metricsBI toolsdashboards · joinsNotion + wikisoft semanticsDocs + Markdownnotes · contextCONTEXT LAYERAuto-built. Reviewed in git.Hard semanticsYAML + SQL the warehouse runsSoft semanticsMarkdown the team readsreviewed like codeAI agentaccurate SQL · governed

A context layer ingests your warehouse, BI tools, and docs into hard and soft semantics, then serves governed SQL to your agent.

Modeling with Analytics AI Agents, with a Context Layer

Every business or data analyst faces mediocre results when prompting Claude or Codex on their data stack. It might figure out some context by reading the git repos or other metadata it can find. Still, the hard part is teaching the internals and business context that are unique to each business domain and company. So, how do we make AI agents for data and analytics reliable and accurate for database queries? Do we need to manually copy and paste Google Docs and Markdown files into the prompts, or can a context layer provide a more reliable, safe, and governed way to ask an AI assistant for analytics?

These days, it’s much easier to ingest or add almost any number of new data sources with custom-built ETL data pipelines, whereas before we had to make hard decisions about what to include and what not. An AI agent, such as Claude or Codex, can merge multiple data pipelines that access the source database via CLIs and destinations via MCP, API, or CLI. But we still need an API and a process for updating source data, not just once. We need to make sure, test, and verify that the data is correct, potentially more than ever.

The challenge remains in modeling the data in a way that represents what the business is, making sure data flows fast but is also correct. But any AI assistant is only as good as the context we give it, and how easily it can read and express context, metrics, models, and knowledge. So, does the context layer solve these problems?

The context layer primarily supports the accuracy of SQL queries, continuous updates to the business context, and governance. Additionally, with a newer context layer, we can include more relevant business insights that are stored internally, usually in unstructured form, in tools like Notion as business documentation. Traditionally, these data weren’t included because they weren’t a “pure data source”, but they can be helpful for decision-making, especially when done in an agentic way. Every bit of human-written data can help the agent make better decisions. So the ultimate problem we solve is to include more useful data and have a more agentic engineered way that is faster and more streamlined, especially to start with.

What’s an Agentic Context Layer?

The core is to turn warehouse metadata, BI tool definitions, query history, docs, and approved metric definitions into reviewable files that agents can search and execute. We feed valuable metadata and actual data to agents and humans so that they can be ingested and discovered more easily in one place.

It’s a further evolution of a [[semantic layer]], such as Cube or AtScale, which are more focused on the modeling of metrics and domain knowledge and usually have less extensive knowledge of documentation, metadata outside of metrics, joins, and source data schema. And building a semantic layer from scratch can take time, as we need to extract metrics and unify them in a single repository in an additional layer. The main driver for a context layer is that it’s automatically generated and includes additional business knowledge that lives outside the technical containers of data engineers, such as data catalogs, DDLs, or YAMLs, in tools like Confluence, Notion, and internal wikis.

I have written about how to model data in the agentic era, Data Modeling for the Agentic Era (Semantics, Speed, and Stewardship), and we’ve seen how metrics and context can strengthen a BI tool or insights. The context layer is all about context that we already have from our database catalogs (information_schema, data types, tables), more technical metadata, but also from our metrics inside the BI tools or in a declarative YAML, while adding new sources such as documentation in Markdown, rich text, fetched via API, CLI, etc. All with the goal of giving agents and LLMs more information to make the right calls autonomously.

Maxime Beauchemin calls documentation, wikis, and Markdown soft semantics, whereas YAML and the SQL metrics are hard facts or hard semantics. More soft semantics, such as business context from domain experts, help agents, and we need to make them available to other agents and humans too, building a full context layer that grows fast, where governance is essential.

Enhancing Gen AI Trustworthiness

Google describes How Looker’s semantic layer enhances gen AI trustworthiness, and a semantic layer can reduce errors for generative AI and autonomous agents creating queries. They say:

With the foundation of a semantic layer, rather than asking an LLM to write SQL code against raw tables with ambiguous field names (e.g., order.sales_sku_price_US), the LLM is empowered to do what it excels at: searching through clearly defined business objects within LookML (e.g., Orders > Total Revenue). These objects can include metadata and human-friendly descriptions (e.g., “The sum of transaction amounts or total sales price”).

We also acknowledge that data modeling for agents is not the same as data modeling for humans, and that shapes how we work with agents and context together.

Context Layers Saving Literal Money

In the end, it’s also saving money, as more governance means the agent knows the system in more detail and therefore needs to make fewer queries to the actual database, or can do so in fewer iterations.

Queries on Snowflake, BigQuery, and Fabric can be expensive, and agents work at a different speed than humans. If we add more context to the agents, they need fewer queries because the information can be extracted from the retrieved context.

The agents have access to all this data, either directly or through CLI, skills, or MCP. The big advantage is that the agents get a warm-up and do not start with a cold database connection. All the warehouse schema, business semantics, BI usage patterns, and human documentation are in one single place.

Also, on the other end, verifying that the queries are correct is hard. More context can help with this.

Entering ktx: Open-source Context Layer

Kaelio launches its new open-source context layer solution called ktx. The goal is to get an OSS context layer solution that everyone can use for the future of analytics.

It started as an AI data agent platform, with the agent performing well on accuracy benchmarks against other platforms, which helped make the case for developing it further. Dashboards and agent interfaces are easier to build than ever, but these data apps need more durable and complete knowledge from all the company’s data. This extended data added with ktx makes the agent more accurate.

[!info] Want to get started immediately?
You can quickstart with a demo project that fetches data from Postgres and Notion, with a dbt project and a Metabase interface. See Quick Start with public demo data, or check out the GitHub repo for the full code. If you already have a project with relevant context and sources, get started with a simple prompt: Run npx skills add Kaelio/ktx --skill ktx and use the ktx skill to install and configure ktx. More at the Quickstart docs.

How Ingesting Business Context Works

So how do we enhance business context for agents to make use of it?

Ingestion happens through two parallel items:

• ingests and prepares both data context such as data models, historical queries, BI dashboards, etc. and
• business context such as internal docs, Slack, etc., in a format that data agents can use to operate more reliably and with stronger governance.

If we look at a high-level overview, compared to a common semantic layer or logical layers in BI tools, you can build a knowledge layer around your business, with not only structured data but also unstructured knowledge from your company-wide internal documentation on Notion, or your data in Markdown or a git repository. ktx helps navigate and maintain this business knowledge.

Both of these can substantially enhance your “context”, basically critical business insights that are usually internal or in a format that can’t be easily used. The challenge here is obviously to separate the updated and accurate documentation or knowledge from the outdated, which needs more focus or rigid process flow/rules on documentation to keep the latest updates, adding an owner and also going back to docs and invalidating them as no longer relevant to improve the context quality.

A knowledge layer wraps structured data and unstructured company knowledge into one place agents can use. See the full video.

Smart agents such as Claude, Codex or any other that get context have built-in tools to understand business metrics, and combined with this newly ingested unstructured knowledge, they can help us get accurate responses against our warehouse or across our various data sources, and act as the human curator of data sets and entities for BI dashboards and apps.

Agents pair their built-in reasoning with ingested unstructured knowledge to answer questions across your sources. See the full video.

If you compare using raw Claude Code vs. the ktx semantic layer component¹ with extended context, you get faster and more correct answers with ktx. Here’s such an example:

ktx has basically two connected sides where the first one builds and maintains the context layer, and the second serves that context to agents at runtime, which you see above.

Anatomy of Context Layer in ktx

The anatomy of a context layer is based on two files and two jobs. YAML for what the warehouse can execute. Markdown for what the team needs to interpret it. Both are committed to git and reviewed like code.

The wiki is the self-organizing collection of ingested Markdown files. And for the YAML files, ktx has a similarly self-organizing executable semantic layer.

ktx ingest pulls raw data from your data stack into a context layer, where you can also edit and create additional context manually. Additionally, you plug ktx into your agent of choice, which allows that agent to send extra memories for ingestion to keep the context up to date based on actual source data.

The semantic part is a set of functions that define metrics precisely, like “monthly revenue” or “profit”, so if you ask your agents, ktx knows exactly how to pull the data and answer the query with SQL, running it consistently.

Context as Code: How to Use Context Layer

ktx is open source and you can install it with npm install -g @kaelio/ktx. It has a self-improving context layer that teaches agents how to query your warehouse accurately, from approved metric definitions, joinable columns, and business knowledge it builds and maintains for you.

With the ktx CLI you get context as code, writing wiki pages and semantic-layer definitions as git-based files you can review, diff, and merge. This is to update and enhance current context (documentation) you have in your company, and provide it to your agents as well. Git-based, so humans and agents can collaborate on it by editing the same context.

So how does it work? Here are the most common commands you can run and use. We don’t go through the whole CLI reference, but cover the most important ones to get a feel for how to work with the ktx CLI. Six commands cover the loop from check status → discover → verify → serve.

[!note] Setup: Demo data
This assumes you have set up ktx and want to work with it. If you want to follow along, or build up your own, check the quickstart with public data Kaelio provides.

1. Is Everything Wired Up? → ktx status

First, get the status, which tells you which connections, LLM, and embeddings are live before you do anything else.

1

ktx status

1
2
3
4
5
6
7
8
9

  LLM           claude-code · sonnet  ✓
  Embeddings    all-MiniLM-L6-v2 (384d)  ✓
  Storage       sqlite (state) · sqlite-fts5 (search)

  Connections (4)
    ✓ orbit      postgres
    ✓ metabase   metabase
    ✓ dbt        dbt
    ✓ notion     notion

2. Discover & Search Metrics → ktx sl

sl is the semantic layer, the YAML definitions that tell agents how to turn “revenue” into correct SQL. Bare ktx sl lists every source and you can add a query to search. Add --json when an agent (not a human) is reading.

1
2
3

ktx sl                       # list all semantic sources
ktx sl "revenue" --json      # search, machine-readable
ktx sl "int_active_contract_arr" --json

The JSON result is rich: a relevance score, a snippet, and matchReasons showing why it matched (lexical, token, semantic). Trimmed:

1
2
3
4
5
6
7

{
  "name": "int_active_contract_arr",
  "path": "semantic-layer/orbit/_schema/orbit_analytics.yaml#int_active_contract_arr",
  "columnCount": 5,
  "score": 0.036,
  "matchReasons": ["lexical", "token"]
}

This is what makes an agent cheaper and more accurate as it finds the right metric from context instead of guessing field names against your warehouse.

3. Verify Before You Trust → validate + query --format sql

When you edit a metric, always test first against the live schema with the following command:

1

ktx sl validate mart_revenue_daily --connection-id orbit

validate catches missing columns, bad joins, and unsafe source names, so you fix the YAML before it hits production data. Then compile the SQL and read it before running anything:

1
2
3
4
5

ktx sl query \
  --connection-id orbit \
  --measure mart_revenue_daily.total_net_revenue \
  --dimension mart_revenue_daily.revenue_date \
  --format sql

Swap --format sql → --execute --max-rows 100 once the SQL looks right. The pattern is always: compile, eyeball, then run.

4. Search the Soft Semantics → ktx wiki

Metrics are the hard facts. The wiki is the soft semantics we were talking about, such as the business definitions, rules, and gotchas in Markdown that agents search for context. Same shape as sl: bare lists, query searches.

1

ktx wiki "new hire"

1
2
3
4
5

  GLOBAL (4 pages)
    #1  new-hire-onboarding-requirements  — what every new hire must know by week one
    #2  orbit-arr-methodology             — contract-first ARR definition + NRR treatment
    #3  orbit-company-overview            — what Orbit sells, plan tiers, workflow
    #4  implementation-handoff-process    — Sales Ops → CS handoff requirements

When sqlite-fts5 is configured, wiki search is hybrid with lexical, token, and semantic (embeddings) lanes combined and ranked together. That’s how an agent finds “ARR methodology” when someone asks about “annual revenue.”

5. The Raw Escape Hatch → ktx sql

Need to peek at the actual rows? ktx sql runs read-only SQL (single SELECT/WITH only) against a connection.

1

ktx sql --connection orbit "select count(*) from orbit_analytics.int_procurement_qualifying_actions"

Most of the time you want ktx sl query, not raw ktx sql. Reach for sql only when you genuinely need rows the semantic layer doesn’t model yet.

6. Hand it to Your AI Agent → ktx mcp + ktx setup

This is what your business users or engineers might use most often, connected with actual agents. Start the MCP server, then wire up your agent of choice:

1
2

ktx mcp start
ktx setup --agents --target claude-code

setup drops a .mcp.json into your project and installs the analytics skill. Next time Claude Code opens in that directory, it picks up the ktx MCP server and starts answering questions with your governed context, calling ktx sl and ktx wiki under the hood instead of hallucinating field names.

1
2
3
4
5
6
7
8

● Skill(ktx-analytics)
  Called ktx 8 times

● Here are the metrics defined in the orbit semantic layer:
  mart_arr_daily         arr                  ARR ($) — contract-first
  mart_revenue_daily     total_net_revenue    gross − credits − refunds
  mart_customer_health   at_risk_customer_count   distinct high-risk accounts
  ...

The agent discovered every metric on its own. No copy-pasted schema, no guesswork.

The Whole Loop

Recapping the full workflow in one loop:

1
2
3
4
5

ktx status              # 1. is it wired up?
ktx sl "revenue"        # 2. discover metrics
ktx sl validate …       # 3. verify against schema
ktx wiki "ARR"          # 4. pull in business context
ktx mcp start           # 5. serve it to your agent

When you’re done, shut the runtime down cleanly:

1

ktx admin runtime stop

Go deeper: Building Context · Writing Context · Reviewing Context · Serving Agents · LLM configuration

The Workflow: From Source to Context Layer

From reading configured source connectors (1) from databases, BI tools, modeling code, docs and notes, to context building and extracting the context (2) of each, to reconciliation (3) of creating new semantics and metrics based on it, to validation (4) and checking references before agents rely on them.

Resulting in a wiki and semantic layer that is referenced and self-improving, and this is how the semantic layer part looks within ktx:

Why a Governed Context Layer Matters

Wrapping up, I know this has been a lot of new information, but I’m very excited about the direction of ktx and integrating logical layers such as metrics and semantic layers with more advanced business context, and with everything being fully open source for you to try out.

But the challenge of natural language for semantic layers being too imprecise is real. A context layer with hard and soft defined configurations will help the conversational interface via the agents tremendously. More context doesn’t always help, but if it’s well maintained and fetched from the actual source or pulling in actual domain knowledge from Notion pages, it’s priceless.

Also, the feature of easily automating commands via CLI and adding or fixing wrong context in a central repository quickly is super helpful. This keeps the human in the loop and provides a consistent way to verify a governed layer that holds the company’s context while using the power of agentic engineering.

If you like this, try ktx with the getting started guide, and star it on GitHub. It’s all open source.

Full article published at Kaelio.com - written as part of my services

My goal is to offer a collection of wisdom, insights, and principles gathered over the years. Giving you an outside view from someone who has used Dagster since back in 2019, used it at enterprise scale but also for my hobby projects (e.g. real-estate project). The piece should give you a holistic view of Dagster’s place in the data ecosystem, how to deal with the complexity of data architecture and enterprises, and scaling your data jobs.

This article shows you how orchestrators such as Dagster are built for an open data platform that integrates the full data ecosystem, with the shift to data assets instead of DAGs, reducing complexity and applying data engineering best practices.

[!note] Definition of Almanack (also spelled “almanac”)
The term refers to a publication containing a variety of information on a dedicated topic. The modern usage of Almanack, particularly in the context of books like those by Charlie or Naval Ravikant, is often metaphorical. It suggests a collection of wisdom, insights, or principles gathered over time.

What is Dagster

In late 2018, on a co-working and co-living sabbatical in Bali, I was searching for something to bring the data warehouse out of the drag-and-drop world of SSIS and Oracle reporting and into a code-first, developer-friendly workflow. I looked at ODE, BiGenius, TimeXtender, and WhereScape, but found that none of them quite fit my open source and programmatic preferences, so I tried to build something myself but didn’t succeed. A year later, back at my 9-to-5 in Copenhagen, I heard Nick Schrock on the Data Engineering Podcast describing the motivation and story behind a Python framework called Dagster that did exactly that. I was hooked, and have used Dagster ever since.

Early Focus on Developer Friendliness

To understand the context of 2019, you must understand that back then, most ETL jobs were triggered with cron or bash scripts, and if there was an error, the only option was to re-run in the next nightly window where production wasn’t touched. Dagster, as explained by Nick in the podcast, focused on developer-friendliness, in particular for ETL developers back then, and that focus hasn’t changed today for data engineers.

So what is Dagster? The original idea, started in 2018 during a sabbatical after Nick worked at Facebook, came with this definition:

One of the goals of Dagster has been to provide a tool that removes the barrier between pipeline development and pipeline operation, but during this journey, he came to link the world of data processing with business processes.

Today the definition hasn’t changed much and reads like this from the Dagster Docs:

Dagster is a data orchestrator built for data engineers, with integrated lineage, observability, a declarative programming model, and best-in-class testability.

The initial definition to “link data processing with business” was the key reason that brought me to it, along with the quality of how the components were implemented. Even more compelling was Nick’s visionary outline for 3-5 years ahead: to make the work of data engineers similar to software engineers, and make their daily life easier.

Biggest Shift Early On

This vision led to many new concepts Dagster originally created, which we take for granted in today’s data work, and shifted the work into a more reliable and useful toolset for data engineers.

Data-aware Orchestration Shift

One of the biggest shifts compared to previous tools and orchestrators was that orchestration was fully data-aware from the very beginning. It tried to understand the heterogeneous complexity that exists at every small to large enterprise company, and thrive in it, supporting the full data engineering lifecycle with its platform and data pipeline capabilities built in.

This gave me a toolkit for building reliable data pipelines out of the box early on, with battle-tested features through its users (open-source) and a quality and thoughtfulness I hadn’t seen before. This was personified by Nick and could be vividly felt in the early interviews, but also in the code that the team produced openly on the repo.

For example, backfilling, restartability, or Spark integrations were open on GitHub, to use, and to adapt to your needs. It was a bit like dbt, but instead of modeling your SQL queries, you’d model your data pipelines and integrate complex data architecture.

This also meant that moving datasets or integrating dependencies is strongly supported, not an afterthought. In Dagster you can use resources to work with Polars, Pandas, Arrow, DuckDB, or anything else to pass datasets, and reference data assets declaratively, or even non-existing ones as Dagster knows to create the assets. Compared to Airflow, where you could load only small data with XCom for the longest time¹, this makes code simpler to understand and maintain.

Resources also decouple storage from compute. You could use Apache Spark locally with a single JAR file and in test use a full-blown cluster, or use MinIO as an S3 interface with simple bucket configuration and in production an S3 server from Amazon. Both are interchangeable without changing pipeline logic, by pure configuration — that’s the beauty of declarative data systems, and Dagster embraces this to this day.

Shift to Data-Asset Based Orchestration

Dagster was super early in shifting from DAGs and operational task-based orchestration to asset-based ones such as BI dashboards, tables, reports, ML models — artifacts the user actually cared about. Everything shifted from an imperative approach to a declarative one like Kubernetes or React, where you define what the dataset must or should have, and Dagster takes care of the implementation logic, mapping it to the configured filesystem, compute engine or cluster.

Now you could quickly describe each pipeline with declarative notations like update: daily or change to update: monthly, or if more advanced, you could define a small sensor logic that checks S3 for updates. Instead of implementing all logic in a data pipeline, we apply the logic directly to the asset and closer to the data, which makes it more transparent and integrated into the full data lineage. When updates happen, we have the full graph, but also a leaner and easier-to-maintain setup.

Or zoomed out - you see the focus on the assets, the tables themselves:

This led later to “Software-Defined Asset” and its mental model where you can define an asset pre-runtime and declare connections to upstream or downstream data assets (e.g. real-time housing prices that we fetch from a webpage that does not exist beforehand). Now we can already build and implement our graph and data lineage without having to physically create the dataset first. Software-defined assets use code to define the data assets and are version-controlled through git and inspectable via tooling. This transparency allows anyone in your organization to understand the canonical set of data assets and reproduce them at any time, and also lays the groundwork for asset-based orchestration.

As Rich Hickey said, the aesthetics of a programming language do not matter, only the outcome. Assets play into that fact. In data engineering terms, it’s not code but data pipelines and DAGs, but what everyone cares about are their outcomes: the data assets.

That led to the shift from working with tasks and DAGs to data assets. That’s the developer-friendliness built in from day one: develop locally and deploy to test and production, with infrastructure and technical implementation decoupled from business logic.

It made Dagster a bit more complex to start with — you need to know more upfront — but since every enterprise hits these data engineering challenges eventually, it’s better to embrace the fact and build for it. The result was an improved developer velocity, but what I noticed, too, was the joy of building reliable data pipelines, equipped with tools that helped me deal with errors, infrastructure, multi-tenancy, data science, big data, and everything thrown at me back then.

Not only single Purpose

When I first introduced Dagster at my previous company, all of a sudden other teams started to take notice and also wanted Dagster for other work such as provisioning infrastructure with one-click deployment. Especially the cloud platform team needed a tool to automate its scripts to deploy on Kubernetes, OpenShift, and everywhere else, but other teams also had needs to automate. With Python as the programming language for Dagster, reading from an external FTP server, transforming the data, and uploading it somewhere via API were not multi-month projects across different teams. Dagster’s flexibility and integration into other tools and systems were a key strength for most teams.

Observability and monitoring were another addition. Every run was logged in the UI and everyone could see the rich metadata of each pipeline run. And because it was open source and had a rapidly growing community, support and ideas didn’t run out.

Dealing with the Complexity of Enterprise Systems

If you have worked at any company larger than 10 people, you have noticed pretty fast that you are dealing with multiple source systems, different CRMs, different ERPs, multiple cloud platforms. Most enterprises have all major cloud platforms running in production, whether it is Amazon services, Google GCP, Azure, or any other major platform. You as the data engineer are the one making sure to integrate them, and basically deal with the complexity that comes with it.

How to Reduce Complexity?

First, acknowledge it: heterogeneous data complexity is a fact of the enterprise data lifecycle. Second, lean on tooling with technical integrations and written code that implements each vendor’s API, so we don’t build everything from scratch repeatedly. Third, work around the actual data assets the users want, not DAGs. With assets we declare outcomes, tests, and dependencies, and the system handles the rest. That cuts dependency hell, an unproductive Software Development Lifecycle, and the fear of change.

Composable is Making Systems Simpler

A more holistic framing comes from Simple Made Easy by Rich Hickey, creator of the Clojure functional programming language, where he debates what makes systems complex: state and objects, lots of vars, syntax, inconsistency. His conclusion is that composable is what makes systems simpler (like in music for a composer, which is what he created Clojure for): the ability to assemble, reassemble, and swap individual components into a flexible whole.

Dagster has exactly that ability, too. It integrates the core principles of Functional Data Engineering — pure and idempotent tasks, immutable partitions, reproducibility, versioning — directly into the framework.

State, Hickey points out, is never simple. Unfortunately for us, data engineering is all state: every datum is tied to a timestamp of when it was created, processed, or backfilled. Fortunately, helping us manage that state is what Dagster does: data assets, virtual assets, partitions, incremental materialization, and partitioned asset checks that evaluate a specific partition of an upstream asset instead of the whole dataset.

Rich concludes “Simplicity is a choice”, echoing Leonardo Da Vinci:

Simplicity is the ultimate sophistication

(Open) Data Platform

But how did the shift and the data engineering principles evolve, and how can we apply these as a unified solution?

With Dagster’s data-aware orchestration, shift to assets, separation of concerns, and multi-use, we are automating harder data and infrastructure deployment problems, and Dagster solves the problem of managing complex data environments more holistically. To me, it feels as though Dagster gave me this peace of mind and the toolset to simplify data engineering in a complex environment early on, and is the right tool for the ultimate sophistication for data work.

Dagster’s approach is composing a data orchestrator that integrates into any type of data work: from data integration with dlt, to transformation with dbt or just Python logic, to updating BI dashboards, to deploying on Kubernetes, all into a unified system. A fully open data platform, making orchestrating data and its flow simpler.

All of these features, combined with DevOps deployment strategies, make Dagster one of the data platform tools that has:

• An integrated UI and control plane for seeing what’s going on, unifying all your tools into a single webpage.
• Lets you see your data assets in a list with extensive metadata: it’s your data catalog showing all tables, BI dashboards, reports, and other data assets.
• Keeps and creates all the metadata when we run data pipelines across all data systems: With end-to-end access, we can also have metadata and data lineage end to end, which helps us understand where the data comes from, and in case of error, where the bad data is.
• Has integrated scheduling, sensors, backfills, and concepts to work with data, built in.
• Supports multi-team isolation through code locations, so different teams can own different parts of the platform without stepping on each other.

As Dagster is open source, it gets promoted from a usual data orchestrator to an open data platform, with the great advantage of transparency: easily patching an error or integration if you need to integrate an obscure system that only your company has, working on the cutting edge with the community, or getting features from them.

[!note] The Venn diagram of Dagster
Obviously you can’t optimize a data platform in all directions. If you look at Dagster as a Venn diagram, it has these three circles: the right abstraction, flexibility, and full automation through programmability.

Control Plane: Center with All Metadata

It integrates multiple different teams such as data engineers, platform and infra teams, with data science and business people who want to run their jobs. Feature-wise, it provides data catalog and contract capabilities, lets you see data assets you’re responsible for and when they last got updated, and shows their downstream and upstream dependencies, all in one real-time observability and monitoring UI.

This is all done through its control plane, which centers all metadata and unifies different data tools along the lifecycle into one platform, something usually only closed-source data platforms achieve. The control plane serves everything in a single view, showing how all processes are working. It’s the operational dashboard for your company.

With data orchestration as the heart of data work, with metadata for any process and access to all source systems we’re pulling from or intermediate systems, it’s in the perfect place to serve as the central metadata store. Think of INFORMATION_SCHEMA, but for overall data work, not only one single database. Only the orchestrator can understand the system and its status this deeply.

Why Open Data Platform: a System that Unifies Open Source

The open in open data platform is interesting, as it’s really hard to build a unifying layer across different stateful data systems, and it’s worth highlighting that Dagster achieved just that.

If we look at open data stack architecture: unlike cloud data platforms that have the same goal, such as Fabric, Snowflake, Databricks, etc., Dagster builds on open standards and is itself an open standard. It’s like a protocol in which we declaratively define our data assets (e.g. Software-Defined Assets, environments, resources) that then get automatically executed with composable computes we define in resources, all interchangeable. Even closed-source engines such as a Databricks Spark cluster work really well.

The hardest part is integrated data governance, lineage, access rights, and compute. An orchestration platform like Dagster doesn’t give you everything, but you get most of what you need, in an open and composable way.

Composable Data Stacks Possible

What this architecture allows is what Wes McKinney calls “Composable Data Stacks” in Monday Morning Data Chat, essentially rebundling the data platform.

Composable data stacks depend on compute engines. With this architecture, plus Dagster resources or external code (Dagster Pipes), we can easily pick and choose what is best suited for the task at hand, not only for different jobs but also depending on different environments. Although there will never be one singular tool for everything, it’s necessary that we have a layer of integration, and there’s no better place than the orchestration layer that separates execution and technical logic from business and already deals with multiple data environments.

Pete Hunt, the CEO of Dagster, said recently:

Our goal is to make AI as lightweight, accessible and cheap as we can to drive enterprise value, while enabling our customers with the infrastructure they need underneath - the orchestration and data platform layer where reliability, support, and recovery actually matter.

Pete is reaffirming that the data platform layer is the priority, so AI can build on top of a great foundation. In the eyes of a data engineer, it’s a dream come true to have an open platform that is declarative to simplify the overall architecture, but also uses AI for the glue code, based on best practices enforced through Dagster as the open data platform.

The Right Abstraction Layer for an Open Data Platform

Charlie Munger’s Almanack distilled decades of investing wisdom into timeless principles and mental models that compound over time. Here I tried the same for Dagster.

The principles touched on won’t be obsolete next year. Data-aware orchestration, declarative assets over imperative DAGs, separation of business logic from infrastructure, and composable stacks with a single control plane are all mental models for building data platforms that hold up whether you’re running DuckDB on a laptop or Spark across three cloud providers.

Eight years after discovering Dagster on a podcast during my time in Copenhagen, I’m still reaching for it whenever a system gets complex enough to need real orchestration. With extensive built-in data quality checks, best practices like unit-testing, local development to prod, separation of business and technical logic, code locations, Domain Specific Languages (DSLs) for non-technical people, pipes and components to run something in Rust or Go, and many more, the Dagster data platform gives you huge leverage building from strong foundations, with the flexibility to change along the way.

It’s the abstraction layer for data engineering to solve hard business problems, an open data platform with opinionated design decisions that compound the longer you build on them.

In the next piece, I’ll get into what it actually takes to operate this — architecture, deployment, and governance — as a follow-on to these principles.

Next Steps

Find Dagster’s official skills for the latest and most updated way of working with Dagster, to feed to your AI agent. Or read the blog post with more information. If using Airflow, migrate from Airflow, or use Airlift for an integration for legacy and critical DAGs still in Airflow.

Find awesome-dagster, and check out further readings of mine at Data Integration as Code: Configuring Airbyte and dbt with Python (Dagster) or Data Orchestration Trends: The Shift From Data Pipelines to Data Products.

Want to use all of this stress-free without the deployment burden? Use dagster+. Great tradeoff between cloud and OSS, still having the OSS Dagster foundation, but profiting from extra features (GitHub integration, cloning, etc.) and not needing to set up a DevOps pipeline or fiddle with Kubernetes.

Full article published at Dagster.io - written as part of my services