Second Brain: Crafted, Curated, Connected, Compounded on

Claude Dashboards: The Jupyter Notebooks of Agent Age

Mon, 12 Oct 2026 10:14:25 +0000

Claude Dashboards were recently announced, and it goes to show that Dashboard are indeed not dead - when Claude has now built their own services.

The integrations is done into these Cloud Data Platform:

Claude Dashboards connects to a data platform like BigQuery, Databricks or Snowflake, or a CRM tool like Salesforce, and is in beta on paid plans.

This is following the trend of dbt Charts, MotherDuck Dives and other BI-as-Code first tools such as Rill Data or Evidence.

# What Claude Dashboards Can’t Do: Disadvantages

It’s like “live data artifacts” but without you being in control.

Christophe Blefari said it best:

You can decide to use Claude Dashboards and build everything on top or you can use the open-source version of it that we are developing. With nao you can do everything you can do with Claude Dashboards and:

managing and self-improving you context layer in one UI versioned with Git

using other LLM providers (even the open-source ones)

having a central governance

Use RLS on your data

self-host it in your own infrastructure

improve the product or even extend it for your own requirements

The nao stands for any BI-as-Code dashboarding tool, you need access to the artifacts to integrate it into the data and open standards. Tightly integrated into Claude will be OK for quick exploration or examples, but not for long-term managed dashboards

# Claude Dashboards Are the Jupyter Notebooks of the Agent Age

Jupyter Notebooks are great for creating a quick exploration or sharing a deep analysis. Still, they are not great for integrating into your data stack or your data pipelines. The same is now with Claude Dashboards.

It’s also a little like Excel, whwre everybody works on their own version. Sure you can share it, but usually everyone want different rules or see different numbers. “Mine is the truth” kind of thing.

# BI-as-Code is Still the Future

Anything code-first is what will win in long-term. It has shown once again; it first started with Markdown with Agentic Skills, Specs that we define with superpowers and others Spec Driven Development (SDD). T

Then Claude went to creating HTML dashboards within Claude, and then today we have full-blown charts. But without its code, or being open source, you are locked-in once again, and you can’t develop or, even more ** important, maintain it over time**. So it just gets stale after a very short time.

A decade ago we discussed no-code vs. code, now it’s in-built in frontier models/apps vs. open source or code-first. Once again, IMO, it’s won by code-first approach, at least in my opinion, and that’s also what Claude Dashboard will need to go towards, or they don’t care about deep BI integration, as BI tools will probably do it better integrated with agnostic agents anyway, so instead, they just try to capture the top of the BI market.

I wrote more about BI is Dead on AI Reveals Why BI Still Matters, about BI-as-Code and the New Era of GenBI and Has Self-Serve BI Finally Arrived Thanks to AI?

# Having a Dedicated Tool vs. the All-encompassing

Also, usually a rule of thumb, at a tool like Claude, that pretends to do it all, is OK if you want to move fast and automate things, but not if you’d like to maintain, or improve, or craft something beautiful or truly important to your business, like Claude Design and others.

I still think a dedicated tool is needed, and I use both today: Claude Code (not design), and then ask it to create an SVG, so I can add and edit it myself. With something like internal dashboards and KPIs, that’s the same. If the insights are truly important, none will just one-shot them and then forget, they need care over time. That’s where I see dedicated BI Tools still winning. Even better, if you use open-source Code-first BI tools, you have the best parts of both.

# Example of Claude Dashboard (Compared to BI Tools)

I asked to recreate an extended version of my Listmonk newsletter dashboard, including all the social media that I built with Rill Data and Evidence to recreate with a Claude dashboard.

# Dashboards in Rill, Evidence and Claude Code

This is how it looks in Rill:

This is how it looks in Evidence

Here’s how it looks with Claude Code:

# The Differences between a BI Tool and Claude Dashboard

In Rill and Evidence, I have all the artifacts locally, which means I can version them in git, I can use an agent to add a measure, collaborate with humans ergo Human in the Loop, and I have everything locally or on a server if checked in.

Rill:

tree -L 2
.
├── connectors
│   └── duckdb.yaml
├── dashboards
│   ├── campaign_analytics.yaml
│   ├── campaign_impact.yaml
│   ├── list_overlap.yaml
│   ├── lists_subscribers.yaml
│   ├── newsletter_health.yaml
...
├── metrics
│   ├── bounce_events.yaml
│   ├── campaign_impact.yaml
│   ├── campaign_summary.yaml
...
├── models
│   ├── api_campaigns.yaml
│   ├── api_daily_activity.yaml
│   ├── bounce_events.yaml
│   ├── campaign_impact.yaml
...

Evidence:

├── evidence
│   ├── access.yaml
│   ├── AGENTS.md
│   ├── CLAUDE.md
│   ├── connection.yaml
│   ├── evidence.config.yaml
│   ├── pages
│   ├── queries
│   └── theme.yaml

Claude Dashboard uploads all its code and artifacts Claude servers, even the data… The first thing it did, was upload all local JSON files, then it ccreatedinteractive dashboards (looks like normal HTML dashboards which I used two years ago with mviz: skills):

├── dashboard
│   ├── charts.js
│   ├── index.html
│   ├── pages.js
│   └── styles.css
├── export_data.sh
├── README.md

Where the artifacts live: artifacts on Claude’s server: 4 page files in its store + 22 attached JSON files. Zero local dependencies to open it. Local files only for:

export_data.sh → refresh data (then I re-upload).
dashboard/ → edit page (then I write to store). Could even drop it; readable back from the store. Kept for git history.

You see, it creates a bunch of CSS and JavaScript code, so it’s not really maintainable. If you don’t need to maintain it, and it is just a PoC, then that’s a great tool to have.

Darragh Murray shows it well too in Claude Dashboards and the shift from chart builder to chart specifier:

So it’s nice to quickly spin up a dashboard with Claude, but it’s not a replacement for any BI Tools, especially not if they are code-first with BI-as-Code.

# Claude Dashboard Artifact Runs only inside Claude

Claude Code says why artifacts can’t be copied locally:

Why

Page files (index.html, pages.js, …) = only half. They call dash.data(), dash.calc(), dash.onData(), dash.colors, theme tokens (--cds-*), d3 global, Sources panel. All come from the type’s runtime (artifact-type/app.js, app.css), served by claude.ai.

Data loads from artifact store + /_blob/... assets via claude.ai APIs, authenticated. No local equivalent.

Could download type’s runtime files, but it talks to claude.ai backend. Not standalone.

Type’s “exporting” path = send to Grafana/Hex/Omni/etc., not local HTML.

Option to use locally: the plain HTML version works, it’s self-contained, but that’s not really what Business Intelligence and dashboards are all about.

It’s more of a report similar to what we did with SQL Server Reporting Services and others, sending a PDF or static statement. A static snapshot: no drill, no refresh, and no shared definitions.

# Good for PoC or Analysis of Your Data

As all data and artifacts (CSS and JSON’s) live on Claude, it’s very hard to maintain. Obviously, this can change, but I see it as providing a service for the TOFU (Top of Funnel): users who don’t necessarily know about BI and just want a “dashboard”, or to explore the data as part of PoC or similar, it’s perfect.

Kuhrt Buhler said, to better call it “Claude Data Analysis”, which is for analyzing data and getting a feel for it; I fully agree.

# Continuing Maintenance: That’s why I Prefer SVG

Besides, all the dashboards will look the same now, similar to Claude Design (if you don’t tweak it heavily), and look all the same. But what I don’t like most is that you get locked in and don’t have the power tools to do it yourself.

I prefer SVG over PNG, the same way I prefer a BI-as-Code first compared to Claude Dashboard—it allows me to create a first fast image/dashboard, and continue updating/edit afterwards the way I like.

# Further Reads

See BI Tools categorized in BI-as-Code and others
Vibe Coding dashboards.
A first look: Claude Dashboards and the shift from chart builder to chart specifier

Origin: Build live dashboards and animate explainers with Claude | Claude by Anthropic
References: Claude Code, Self-Service BI

Different Types of Writers

Fri, 09 Oct 2026 21:56:39 +0000

Here I collect the different types of writer I came across so far.

# Two Types of Writers

There are at least two types of writers:

The one that outlines first, and then fills in the information
1. This is the The Architect
The one that has a vague idea of what to write about, maybe a title, and then goes with the flow and sees where it takes him/her
1. Like Stephen King, having fun on exploring. This is The gardener to Brandon Sanderson in The Tim Ferriss Show. See also more in Type of Notetakers

# Writing Process

My Writing Process and everyones writer, see How I Write (Podcast), writes differently. Check out mine summarized here in Knowledge Management in the Digital Age: From Zettelkasten to Startup Owner | Obsidian Workflow as presented as part of My Gigs.

# What’s Your Choice?

How do you write? What’s your medium (paper, Markdown, proprietary format)?

What Archetypes of Notetakers are you, an architect, gardener, or librarian? What app do you use? What keyboard? What font?

It might not make all the difference on its own, but together it does. Choose wisely.

# Different Notes and Notetakers

There are also other Type of Notes that can be written and other Type of Notetakers. Which might help to organize your knowledge.

I think it’s important to have a place where you store your knowledge (notes from books, insights you read online, ideas, etc.) that can help you when you write too.

# Structure of Writings

# FAQ

# Writing vs. Typing

The questions of Curious about WriterDeck philosophy: Writing vs. Typing? : r/writerDeck triggered me to think a bit more.

# Outline vs Chain of Thoughts

Origin: My Gigs / Curious about WriterDeck philosophy: Writing vs. Typing? : r/writerDeck
References: My Writing Voice, Structure of Writing

SQLMesh

Fri, 09 Oct 2026 21:37:06 +0000

Don’t hack custom scripts or use half-baked tools. SQLMesh ensures accurate and efficient data pipelines with the most complete DataOps solution for transformation, testing, and collaboration

Similar to dbt but tries to understand more of the semantics of the SQLs, where dbt is just stitching together blobs of SQL, SQLMesh tries to understand the SQL statements more. That allows them to do the translation from Duckdb-SQL to BigQuery-SQL, because of that awareness.

Blog about it: Tobiko Data - Efficient Development with the SQLMesh Browser UI

From GitHub - TobikoData/sqlmesh: Efficient data transformation and modeling framework that is backwards compatible with dbt.

# Docs

twitter.com/Captaintobs/status/1653486832590684161

# Plans (Environments)

SQLMesh concepts with plans that apply to different environments (prod, dev) are elegant. Even fetchdf is integrated into the CLI.

Also, on the right, you see SQLMesh auto-detecting the new columns as non-breaking and simply applying the (virtual) changes y. x.com

# Virtual Environments

Check more at Virtual Data Environments.

# They Have Column-Level Lineage

Tobiko Data - Column level lineage for dbt

Also, comments can be lineaged, with just adding a comment on previous column.

# Installation on top of dbt

Using SQLMesh on dbt Projects for Column Level Lineage 🚀-

# Semantic Understanding

SQLMesh actually understands the SQL you write and improves developer productivity by finding issues at compile time. Built-in column-level lineage provides a deeper understanding of your data model and transpilation makes it easy to run your SQL across multiple engines.

More on Semantic SQL and Semantic Models.

# Using SQLGlot Expressions

SQLMesh automatically parses strings returned by Python macro functions into SQLGlot expressions so they can be incorporated into the model query’s semantic representation. Functions can also return SQLGlot expressions directly.

For example, consider a macro function that uses the BETWEEN operator in the predicate of a WHERE clause. A function returning the predicate as a string might look like this, where the function arguments are substituted into a Python f-string:


`from sqlmesh import macro @macro() def between_where(evaluator, column_name, low_val, high_val): return f"{column_name} BETWEEN {low_val} AND {high_val}"`

The function could then be called in a query:


`SELECT a FROM table WHERE @between_where(a, 1, 3)`

And it would render to:


`SELECT a FROM table WHERE a BETWEEN 1 and 3`

Alternatively, the function could return a SQLGLot expression equivalent to that string by using SQLGlot’s expression methods for building semantic representations:


`from sqlmesh import macro @macro() def between_where(evaluator, column, low_val, high_val): return column.between(low_val, high_val)`

The methods are available because the column argument is parsed as a SQLGlot Column expression when the macro function is executed.

Column expressions are sub-classes of the Condition class, so they have builder methods like between and like.

See more at SQLMesh macros - SQLMesh.

# Integrations

# Features

A thoughtful write-up from Reddit: SQLMesh versus dbt Core:

If you want my honest opinion, these are not the main reasons to use SQLMesh (some of them are indirectly, but a lot of them don’t matter that much, especially for smaller teams / data), but there are a lot of other great reasons outside of this:

Python macros (better than jinja + they are testable and type checkable)

Compilation is more than just basic ref and jinja parsing but actually validates the SQL, reducing the likelihood of invalid SQL making it out of dev and into production (you should have a staging environment etc… ideally, but we’ve all worked in some nasty situations where we don’t for some reason + even with that, it’s nice to be able to push out a change without having as much fear regarding stupid mistakes)

Breaking changes and potential needs for backfills are made obvious to users in cases where it may not be so obvious (e.g., changing value of a column or adding a duplicate column name in table ‘a’ upstream and having a “SELECT a., b.” in a downstream model - this would not get caught in DBT)

CTEs can be unit tested (this is awesome for models that are large and have a lot of complex SQL) + if you choose to use mostly or only ‘standard’ (sqlglot convertible) SQL, then for many cases you can actually run almost all of your tests using DuckDB rather than hitting the warehouse

Native table diffing between prod and dev (extremely convenient and extremely valuable for understanding the effects of your change between prod and dev environment). It even works between different engines / warehouses which is awesome for migrations (never got the chance to use it, but I certainly would have liked to have had it for my recent one from redshift to snowflake)

Native batching for backfills - DBT recently added something like this, but it’s not as robust, and I can attest to having fought with DBT in this in the past when needing to backfill a model for 1+ year and wanting to pick the bucket size to be like 7 days at a time and not one mega query of 2 years backfill in one go, which may not be possible (especially in things like bigquery where you can’t just increase the warehouse size for a backfill)

Native understanding of partitioning. Just makes life easy when working with something like Dagster, especially if you are dealing with lots of different types of partitioning (especially intraday). It’s handling of backfills and making sure that models have ran completely means you can just run it and know that once it is done, that everything is up to date and complete (rather than having to do a bunch of extra hacky checks all the time, which are just a lot of mental overhead and boilerplate which are often more error prone in my experience)

SQLMesh also seems en route to start supporting multi-engine repos for things that have the same catalog, which means that potentially if you had an iceberg catalog perse, then you could have a traditional warehouse + Trino for example which could open up huge number of possibilities for federated queries (this is very new and still needs to be ironed out, but something like this hasn’t even begun to be brought up seriously on the DBT core side to my understanding)

# History of SQLMesh

# Roadmap after Acquisition

Yuki Kakegawa as one of the maintainer says:

So we don’t have the product roadmap just yet, we’re trying to get more contributors just in general and increasing our rate to response, review and close PRs/issues at the moment.

But what product development we want to focus on for the long run, we’re actively discussing and if you have any thoughts we’d love to hear it!

The development pace def declined but sqlmesh is in a good, hopeful position now.

As for the specifics for the product roadmap items, what we’ve been discussing is:

Vertical depth: Go deeper on what sqlmesh is already good at

Horizontal integration: get more tools to build sqlmesh integrations. Or use the dbt ecosystem somehow. And implementation of low hanging fruit features like sqlmesh docs etc

Comment on Reddit to speculate what’s the future of SQLMesh:

It is a curious move indeed, in a way I feel sad to see a great competitor go through a path like this.

I guess it was kinda obvious that Fivetran would not end up running both dbt and swlmesh as part of their product, and it does seem like sqlmesh was used as leverage for the dbt acquisition by Fivetran. Fivetran could have invested further into sqlmesh to make it a bigger and stronger competitor to dbt, and they chose not to.

The way I read the situation now is that sqlmesh will be developed outside the Fivetran umbrella. It could be utilized to get a bit more friendly vibe than Fivetran itself since the community don’t seem to be a big fan of them, and try to get in anywhere while they can. Another alternative could be that Fivetran leadership had to make a decision on what to prioritize and they might have picked dbt over sqlmesh. I have also noticed some important members of the team leaving Fivetran recently, which signals a similar pattern.

Regardless, I have utmost respect for Toby and the team for what they have built, they have definitely pushed the space forward and contributed great ideas. Looking forward to seeing what they’ll do next. Sqlmesh joined linux foundation . What it means : r/dataengineering

# Alternatives

# Dbt Vs SQLMesh

Just some notes and comments regarding dbt (and dbt fusion) vs. SQLMesh as discussed:

dbt just announced a new closed-source version of it, called dbt Fusion (i have some notes here). Basically dbt core (the open source version) will probably be unmaintained over time by dbt as all new features go into the dbt fusion engine, which needs a paid dbt cloud.
dbt fusion is similar to SQLMesh, Rust based, super fast and does compiling features such as checking data types, or data loads ahead of runtime where as the dbt core does not have these features and you have to run the pipeline to get the errors (more expensive, takes longer)
Due to that, SQLMesh has a great momentum. They also created and use SQLGlot (open-source transpiler) that can read almost any SQL syntax and translate in the others.
SQLMesh: Unless you use the cloud version of them, it’s very CLI heavy, which is good if you want to run it in Kestra or embedded, but less if you want to have people working on it. But they column-level lineage, plans for environments and semantic understanding that is quite sophisticated.

I have used dbt much more than SQLMesh, and dbt is for sure the market leader. But if you start out, it’s a really good idea to check out SQLMesh too, as you are doing.

Origin: dlt and Data Engineering Podcast: Eliminate The Overhead In Your Data Integration With The Open Source dlt Library

Photoshop Alternatives

Fri, 09 Oct 2026 14:13:12 +0000

Adobe Photoshop is the most photo edit tool, but the subscription and the company behind is one of the worst. So people have been searching for alternatives since then.

# Free Adobe Copies for Mac, Windows and Linux

These are free and available on Windows, Mac AND Linux:

PhotoCraft (Photoshop): The image editor you already know how to use.

Not Photoshop, but other Adobe replacements:

VectorCraft (Illustrator)
FilmCraft (Premiere Pro)
LightCraft (Lightroom): Your photos. Your pixels. Your machine.
PdfCraft (Acrobat)
EffectCraft (After Effects)
DesignCraft (InDesign)

Apps overview by ArtCraft, the code on GitHub, and a great video showcasing it, still early days but already very useable: Somebody Vibe Coded EVERY SINGLE ADOBE App - YouTube

# LightCraft

# Lightroom Alternatives

Above LightCraft
Rawmakase: Free, fast, Lightroom-compatible RAW photo editor for Linux, macOS, and Windows.
Rawnd: Cull and develop RAW photos on any computer, in your browser.

# Further Reads

My Journey to Immich - Self-Hosted Photo Management | Alex’s Blog

Origin:

Listmonk

Fri, 09 Oct 2026 13:32:13 +0000

Listmonk is an open-source and self-hosted newsletter and mailing list manager. Performance and features packed into a single binary. Free and open source.

My new Newsletter is published with Listmonk on list.ssp.sh. So far I used Mailchimp. I also liked Buttondown, unfortunately, it costs 9$ for above 100 subscribers.

By using open source Listmonk, I have the sending emails and collecting and unsubscribing from a newsletter, which I needed. But I had to deploy it myself. I used Railway.app.

# Settings

# Automatic Confirm Subscription Email by Clicking in Email

Default behaviour, if you click on the confirm button in your email:

It wil infact not confirm yet, you need another click once again confirm on the linked page. I believe this is why many have not confirmed my email subscribtion, tough many are subcribing itself.

There is a workaround for this by pasting the following piece of JavaScript in Settings -> Appearance -> Public -> Custom Javascript:

`1`	`document.querySelector(".optin-form").submit();`

After that, the above confirm will open a browser window and automatically click/submit the button after opening the browser. I think this should be default as it won’t work if JavaScript it blocked in your browser.

With latest version, this can be now set

, see Confirm subscription on opt-in email link click · Issue #556 · knadh/listmonk

# Welcome Emails

Listmonk does not support automatically sending welcome emails or other messages. It is better to use Email Automation, something like Mautic, as mentioned in the GitHub issue.

A hacky solution with Listmonk on Auto reply on subscription (Issue #206).

The alternative I use for now is PayHip, see on My Services.

# Opt-In Campain to None-confirmed Subscribers

See export and import at Memberstack

# Listmonk Analytics (Views + Clicks)

Listmonk has no global “track everything” switch, there’s two mechanism, one for views on campain level, and one on links clicked inside the campain, both mechanisms are opt-in:

{{ TrackView }} is needed in template and will show in tracking for views. Add once to the campaign template, right before </body>. Renders as an invisible 1×1 pixel (https://<listmonk-url>/campaign/{campaign}/{subscriber}/px.png); loading it logs a view. Requires images enabled in the recipient’s mail client, and root_url must be publicly reachable.
@TrackLink — link clicks. Appended per link, at the end of the URL: [text](https://example.com@TrackLink) (Markdown) or <a href="https://example.com@TrackLink"> (HTML). Listmonk rewrites it to https://<listmonk-url>/link/{link}/{campaign}/{subscriber} which logs the click, then redirects. Never append to {{ UnsubscribeURL }} & friends — breaks the link.
- This can be done on user level or general:

General Settings → Privacy → “Individual subscriber tracking”: only controls attribution, not on/off. On = views/clicks tied to the subscriber; off = counted anonymously. Tracking itself is enabled purely by having the tags in template/content.

Results: campaign → Stats (per campaign) and the Analytics page.
Caveat: opens are fuzzy (image-blocking undercounts, Apple Mail preload overcounts), trust clicks over views.

# Automation

Sending it with My Amazon SES.

Make sure you set up and verify the domain, and set some other settings like SPF, DKIM, and DMARC records. Check Bulk or mass email service for Newsletter and Email Automation.

I think I used these checklists:

Tutorial: Setting up DMARC, SPF, DKIM with Amazon SES - EmailOctopus knowledge base
- (backup: Setting up DMARC, SPF, DKIM with Amazon SES)
- Alternative: Site Unreachable
Amazon docs:
- Email authentication methods - Amazon Simple Email Service
  - Complying with DMARC authentication protocol in Amazon SES - Amazon Simple Email Service
  - Authenticating Email with SPF in Amazon SES - Amazon Simple Email Service
Set-up bouncing with Amazon SES
- Bounce processing - listmonk / Documentation (internal

Sending with AWS SES cost a little

Although you pay, it’s almost neglectable, at least if you don’t have millions of subscribers.

# RSS

Based on RSS new email - Provide an RSS integration for listmonk, a self-hosted newsletter and mailing list manager. GitHub - listmonk-rss: by Stephan Heuel.

I just enabled automatic email send when I create a new article based on my RSS-Feed. I used Above repo by Stephan and forked it. Locally I can run make dry_run or live:

1
2

cd ~/Documents/git/sspaeti.com/listmonk-rss
make create_campaign

It works like a charm. This will schedule a campaign with the latest articles since LAST_UPDATE with a delay of 180 minutes.

# This is how it Looks

Make command:

New created campain on Listmonk:

And updated LAST_UPDATE on GitHub:

And the Email:

Wdyt? :)

See How to automate Newsletter based on SecondBrain and Bluesky changes.

# Integration into My Email Client

I’ve built my own email client Neomd, and with sending an email to listmonk@ssp.sh it will automatically send a campain to all my newsletter.

See details at Listmonk Newsletter Integration – neomd.

# Interface

# Querying

`1`	`subscribers.created_at >= '2025-09-23 00:00:00'`

`1`	`subscribers.created_at::DATE = '2025-09-23'`

# Subscribe with `Curl`

I added small wrapper around Listmonk to subscribe via curl with curl https://sub.ssp.sh/you@example.com or get the information about `curl -L https://sub.ssp.sh/why

Example:

❯ curl https://sub.ssp.sh/your@email.com
Almost there -> CHECK YOUR INBOX TO CONFIRM.

------

Why subscribe? curl https://sub.ssp.sh/why

Check it out 🙂.

And why to subsribe?

See Go-code here: listmonk-curl, a curl-friendly newsletter signup shim in front of Listmonk.

# Tech Setup and Maintenance

# Upgrade Listmonk on Railapp

Head to your dashboard, and select your Listmonk project.
Select the GitHub deployment service.
In the Deployment tab, head to the latest deployment, click on the three vertical dots to the right, and select “Redeploy”.

see more on Upgrade - listmonk / Documentation

pg_dump (private)
Backup tables (private)

# Backup Postgres DB

Check Automated PostgreSQL Backups.

Restoring data from the backup

Our data should is now being automatically backed up to our S3 bucket. If we ever want to restore this data to a database, we can do so very easily using pg_restore.

First, unzip the backup which is stored as a tar.gz file and then run:

`1`	`pg_restore -d 'DATABASE_URL' BACKUP_FOLDER`

In the above snippet, replace the DATABASE_URL with the connection URL of the database you’d like to restore the data to and the BACKUP_FOLDER with the name of the folder to which you extracted your backup zip file and voila, your backup has been restored!

# Caddy: CORS Settings - Railapp Migration

I wanted to add my newsletter (list.ssp.sh) to my website (ssp.sh), as these are not the same domain, I got a CORS error.

Luckily the new Railway.app template (see here) includes a Caddy settings and this can be easily set. But you need to migrate to the new one.

Here are the changes it’s made - and you see my comment of what you need to change in your variable to make it work - add support for cors · railwayapp-templates/listmonk@39d3e53 · GitHub

The new caddy-template removes config.sh where ENVs where set - so after the migration it won’t wor. You need to add additional variable (or replacing once) to the Railway variables;

LISTMONK_db__host="${{Postgres.PGHOST}}"
LISTMONK_db__port="${{Postgres.PGPORT}}"
LISTMONK_db__user="${{Postgres.PGUSER}}"
LISTMONK_db__password="${{Postgres.PGPASSWORD}}"
LISTMONK_db__database="${{Postgres.PGDATABASE}}"
LISTMONK_db__ssl_mode="disable"

After adding these (I removed the old PG_* and DATABASE_URL) and then it worked as before. Also, if you want to add CORS accepted URLs, you can just add to the above variables - such as:

1
2

LISTMONK_ORIGIN_0="https://www.my-web-page.com"
LISTMONK_ORIGIN_1="https://another-webpage.ch"

# Analytics: I Created My Own Dashboard Based of Postgres DB

Check out the GitHub repo listmonk-dashboard : Dashboard of unique visitors, opening rate, etc with Rill and Evidence. All locally.

# Opening Rate

# List and Subscribers

And more, see GitHub - sspaeti/listmonk-dashboard: Dashboard of unique visitors, opening rate, etc with Rill and Evidence. All locally. · GitHub.

# Listmonk vs. Alternatives

# Listmonk vs. Substack

Fast network growth can fade as fast as it came. Sometimes I wonder if, when most land in email inboxes, and you get 10 or 20 of them, people are still reading them as much as Substack wants us to believe.

And I’ve been there, coming from Mailchimp, then Medium in a way, now Substack, what’s next? The cost of switching is real, too. I love Listmonk if you like to own your list and enjoy a little bit of setup. It’s great. The best part, and why I chose it in the first place, is Markdown support. No code converting from text to an image, or afraid of losing formatting from your text editor to the Substack editor. My LinkedIn Comment

# Further Reads

Open Subscription Platforms
Email Automation
Building an Independent Creator Business with Subscriptions

Origin: Newsletter
References: listmonk - Free and open source self-hosted newsletter, mailing list manager, and transactional mails, Send Checklist for Reaching Audience

BI Tools

Fri, 09 Oct 2026 09:43:38 +0000

Tools of Business Intelligence, there are many.

# Tools

Here a none complete list divided in different categories.

# BI-as-Code

Rill Developer: Fast, BI-as-code dashboards defined in YAML + SQL with an embedded DuckDB/ClickHouse OLAP engine for sub-second queries.
Evidence: BI as Code with a Markdown notebook style, rendering SQL-backed, NYT-style data reports as a static site from plain Markdown files.
Lightdash: Open-source BI that lives directly on top of your dbt project, turning dbt YAML models and metrics into a self-serve analytics layer.
Observable Framework: Open-source static-site generator for data apps, using polyglot data loaders (any language) to precompute snapshots for instant-loading dashboards.
Shaper: Visualize and share your data. All in SQL. Powered by DuckDB.

# Early Open-Source SQL-First Dashboarding

Apache Superset: Open-source data exploration platform born at Airbnb, connects to any SQL speaking engine with 40+ visualization types and petabyte-scale query capabilities.
Metabase: The “5-minute setup” open-source BI beloved by startups, pairing a no-code question builder with a SQL editor and friendly UX.
Redash (dashboarding on top of Delta Lake): Lightweight open-source SQL-first tool — a browser-based query editor over 35+ data sources that turns queries into shareable charts and dashboards.

# Enterprise Self-Service Platforms

Power BI: Microsoft’s BI platform with deepest Excel/Office 365 integration, DAX formula language, and Copilot AI baked into the broader Microsoft Fabric stack.
Tableau: The visual analytics pioneer built on patented VizQL technology that turns drag-and-drop actions into database queries, now part of Salesforce.
Qlik: Known for its patented Associative Engine that in-memory-indexes all relationships so users can freely explore in any direction — including “unrelated” data.
TARGIT: Danish BI suite (Decision Suite) focused on industry-specific, out-of-the-box analytics for verticals like automotive dealerships, heavy equipment, and airports.

# Semantic-Layer-First

Looker: Google Cloud’s enterprise BI where LookML defines a governed, code-based semantic layer that’s now exposed to Tableau, Power BI, and AI agents.
Holistics: Analytics-as-code BI with its own AMQL semantic language, Git version control, and canvas-style narrative dashboards — a modern Looker alternative.
AtScale: Universal semantic layer that speaks SQL, MDX, and DAX natively, bringing OLAP-cube-style modeling (à la SSAS) to cloud warehouses without moving data.
Steep: Stockholm-based metrics-first BI where everything starts from a centrally defined metric catalog, built natively on the semantic layer (dbt Cloud, Cube).
Omni: Founded by ex-Looker folks, combines a governed semantic layer with spreadsheet-like and SQL workflows so analysts and business users share one tool.

# Analyst & Spreadsheet Workspaces

Sigma (BI): Warehouse-native BI with a familiar spreadsheet UI over Snowflake/BigQuery/Databricks, plus Input Tables that let users write back to the warehouse.
Mode: Collaborative analyst-first platform combining SQL, Python, and R notebooks with reports, now part of ThoughtSpot.
Count: A collaborative business intelligence (BI) platform built around an digital canvas like Miro.

# Embedded Analytics & Data Storytelling

whaly: French self-service BI platform positioned as a cheaper Looker alternative, focused on embedding analytics into CRMs, Slack, and field-sales apps.
Toucan Toco: French “data storytelling” platform that prioritizes narrative-driven, mobile-first dashboards for non-technical business audiences.

# Observability & Search Analytics

Kibana: Visualization front-end for Elasticsearch, purpose-built for log, metric, and trace analytics inside the ELK stack rather than traditional BI.

# Agent-Native BI

Graphene: BI built for agents.

# Visualization Frameworks and Libraries

Mosaic: An extensible framework for linking databases and interactive views.
MotherDuck Dives: An interactive visualizations you create with natural language, directly on top of your data in MotherDuck.
Prefect Prefab
dbt Charts
Claude Dashboards: Now (2026-10-08) also Claude building their own: Build live dashboards and animate explainers with Claude | Claude by Anthropic
- Claude Dashboards connects to a data platform like BigQuery, Databricks or Snowflake, or a CRM tool like Salesforce, and is in beta on paid plans.
see more at MotherDuck Dives

# Overview

Source

source

Origin:

Kanagawa

Thu, 08 Oct 2026 11:36:18 +0000

It’s my favorite Color Theme, it’s a mix between gruvbox and tokyonight!, announced on Reddit.

I even created my own Obsidian Theme.

# Color Palette

(from GitHub - rebelot/kanagawa.nvim: NeoVim dark colorscheme inspired by the colors of the famous painting by Katsushika Hokusai.)

Name	Hex	Usage
fujiWhite	`#DCD7BA`	Default foreground
oldWhite	`#C8C093`	Dark foreground (statuslines)
sumiInk0	`#16161D`	Dark background (statuslines and floating windows)
sumiInk1	`#1F1F28`	Default background
sumiInk2	`#2A2A37`	Lighter background (colorcolumn, folds)
sumiInk3	`#363646`	Lighter background (cursorline)
sumiInk4	`#54546D`	Darker foreground (line numbers, fold column, non-text characters), float borders
waveBlue1	`#223249`	Popup background, visual selection background
waveBlue2	`#2D4F67`	Popup selection background, search background
winterGreen	`#2B3328`	Diff Add (background)
winterYellow	`#49443C`	Diff Change (background)
winterRed	`#43242B`	Diff Deleted (background)
winterBlue	`#252535`	Diff Line (background)
autumnGreen	`#76946A`	Git Add
autumnRed	`#C34043`	Git Delete
autumnYellow	`#DCA561`	Git Change
samuraiRed	`#E82424`	Diagnostic Error
roninYellow	`#FF9E3B`	Diagnostic Warning
waveAqua1	`#6A9589`	Diagnostic Info
dragonBlue	`#658594`	Diagnostic Hint
fujiGray	`#727169`	Comments
springViolet1	`#938AA9`	Light foreground
oniViolet	`#957FB8`	Statements and Keywords
crystalBlue	`#7E9CD8`	Functions and Titles
springViolet2	`#9CABCA`	Brackets and punctuation
springBlue	`#7FB4CA`	Specials and builtin functions
lightBlue	`#A3D4D5`	Not used
waveAqua2	`#7AA89F`	Types
springGreen	`#98BB6C`	Strings
boatYellow1	`#938056`	Not used
boatYellow2	`#C0A36E`	Operators, RegEx
carpYellow	`#E6C384`	Identifiers
sakuraPink	`#D27E99`	Numbers
waveRed	`#E46876`	Standout specials 1 (builtin variables)
peachRed	`#FF5D62`	Standout specials 2 (exception handling, return)
surimiOrange	`#FFA066`	Constants, imports, booleans
katanaGray	`#717C7C`	Deprecated

# My Obsidian Kanagawa

See latest

See latest on obsidian_kanagawa/theme.css at main · sspaeti/obsidian_kanagawa · GitHub

:root
{
    /* DARK THEME */
    /* Kanagawa from https://github.com/rebelot/kanagawa.nvim */
    /* Bg Shades*/
    --sumiInk0      : #16161D;
    --sumiInk1b     : #181820;
    --sumiInk1      : #1F1F28;
    --sumiInk2      : #2A2A37;
    --sumiInk3      : #363646;
    --sumiInk4      : #54546D;

    /* Popup and Floats*/
    --waveBlue1     : #223249;
    --waveBlue2     : #2D4F67;

    /* Diff and Git*/
    --winterGreen   : #2B3328;
    --winterYellow  : #49443C;
    --winterRed     : #43242B;
    --winterBlue    : #252535;
    --autumnGreen   : #76946A;
    --autumnRed     : #C34043;
    --autumnYellow  : #DCA561;

    /* Diag*/
    --samuraiRed    : #E82424;
    --roninYellow   : #FF9E3B;
    --waveAqua1     : #6A9589;
    --dragonBlue    : #658594;

    /* Fg and Comments*/
    --oldWhite      : #C8C093;
    --fujiWhite     : #DCD7BA;
    --fujiGray      : #727169;
    --springViolet1 : #938AA9;

    --oniViolet     : #957FB8;
    --crystalBlue   : #7E9CD8;
    --springViolet2 : #9CABCA;
    --springBlue    : #7FB4CA;
    --lightBlue     : #A3D4D5; /* unused yet*/
    --waveAqua2     : #7AA89F; /* improve lightness: desaturated greenish Aqua*/

    --springGreen   : #98BB6C;
    --boatYellow1   : #938056;
    --boatYellow2   : #C0A36E;
    --carpYellow    : #E6C384;

    --sakuraPink    : #D27E99;
    --waveRed       : #E46876;
    --peachRed      : #FF5D62;
    --surimiOrange  : #FFA066;
    --katanaGray    : #717C7C;

    /* Gruvbox from https://github.com/insanum/obsidian_gruvbox */
    --dark0-hard:     #1d2021;
    --dark0:          #282828;
    --dark0-soft:     #32302f;
    --dark1:          #3c3836;
    --dark2:          #504945;
    --dark3:          #665c54;
    --dark4:          #7c6f64;

    --gray:           #928374;

    --light0-hard:    #f9f5d7;
    --light0:         #fbf1c7;
    --light0-soft:    #f2e5bc;
    --light1:         #ebdbb2;
    --light2:         #d5c4a1;
    --light3:         #bdae93;
    --light4:         #a89984;

    --bright-red:     #fb4934;
    --bright-green:   #b8bb26;
    --bright-yellow:  #fabd2f;
    --bright-blue:    #83a598;
    --bright-purple:  #d3869b;
    --bright-aqua:    #8ec07c;
    --bright-orange:  #fe8019;

    --neutral-red:    #cc241d;
    --neutral-green:  #98971a;
    --neutral-yellow: #d79921;
    --neutral-blue:   #458588;
    --neutral-purple: #b16286;
    --neutral-aqua:   #689d6a;
    --neutral-orange: #d65d0e;

    --faded-red:      #9d0006;
    --faded-green:    #79740e;
    --faded-yellow:   #b57614;
    --faded-blue:     #076678;
    --faded-purple:   #8f3f71;
    --faded-aqua:     #427b58;
    --faded-orange:   #af3a03;


    /* LIGHT THEME */
    /* Called Kanagawa Lotus - Colors */
    --lotusInk1: #545464;
    --lotusInk2: #43436c;
    --lotusGray: #dcd7ba;
    --lotusGray2: #716e61;
    --lotusGray3: #8a8980;
    --lotusWhite0: #d5cea3;
    --lotusWhite1: #dcd5ac;
    --lotusWhite2: #e5ddb0;
    --lotusWhite3: #f2ecbc;
    --lotusWhite4: #e7dba0;
    --lotusWhite5: #e4d794;
    --lotusViolet1: #a09cac;
    --lotusViolet2: #766b90;
    --lotusViolet3: #c9cbd1;
    --lotusViolet4: #624c83;
    --lotusBlue1: #c7d7e0;
    --lotusBlue2: #b5cbd2;
    --lotusBlue3: #9fb5c9;
    --lotusBlue4: #4d699b;
    --lotusBlue5: #5d57a3;
    --lotusGreen: #6f894e;
    --lotusGreen2: #6e915f;
    --lotusGreen3: #b7d0ae;
    --lotusPink: #b35b79;
    --lotusOrange: #cc6d00;
    --lotusOrange2: #e98a00;
    --lotusYellow: #77713f;
    --lotusYellow2: #836f4a;
    --lotusYellow3: #de9800;
    --lotusYellow4: #f9d791;
    --lotusRed: #c84053;
    --lotusRed2: #d7474b;
    --lotusRed3: #e82424;
    --lotusRed4: #d9a594;
    --lotusAqua: #597b75;
    --lotusAqua2: #5e857a;
    --lotusTeal1: #4e8ca2;
    --lotusTeal2: #6693bf;
    --lotusTeal3: #5a7785;
    --lotusCyan: #d7e3d8;


}

.theme-dark
{
    --background-primary:         var(--sumiInk1);
    --background-primary-alt:     var(--sumiInk1);
    --background-secondary:       var(--sumiInk1b);
    --background-secondary-alt:   var(--sumiInk2);
    --text-normal:                var(--fujiWhite);
    --text-faint:                 var(--fujiGray);
    --text-title-h1:              var(--neutral-red);
    --text-title-h2:              var(--neutral-orange);
    --text-title-h3:              var(--neutral-yellow);
    --text-title-h4:              var(--neutral-green);
    --text-title-h5:              var(--neutral-aqua);
    --text-title-h6:              var(--neutral-blue);
    --text-link:                  var(--springBlue);
    --text-a:                     var(--dragonBlue);
    --text-a-hover:               var(--peachRed);
    --text-a-url:                 var(--dragonBlue); /*hover internal link, graph view link and standalone links*. External Links no*/
    --text-mark:                  rgba(34, 50, 73, 0.4); /* waveBlue1 */
    --pre-code:                   var(--sumiInk0);
    --text-highlight-bg:          var(--waveBlue1);
    --interactive-accent:         var(--dragonBlue); /* vim cursor background AND highilight on right*/
    --interactive-before:         var(--sumiInk4);
    --background-modifier-border: var(--dragonBlue);
    --text-accent:                var(--autumnGreen);
    --interactive-accent-rgb:     var(--springGreen);
    --inline-code:                var(--springGreen);
    --code-block:                 var(--fujiGray);
    --vim-cursor-backgroung:      var(--fujiWhite);
    --vim-cursor-color:           var(--sumiInk1);
    --text-selection:             rgba(34, 50, 73, 0.5); /* waveBlue1 */
    --text-tag:                   var(--springViolet1);
    --task-checkbox:              var(--sumiInk4);
    --table-header:               hsl(0, 0%, 14%);
    --table-row-even:             hsl(0, 0%, 17%);
    --table-row-odd:              hsl(0, 0%, 20%);
    --table-hover:                var(--sumiInk2);
    --bold-color:                 var(--light1);
}

.theme-light {
    --background-primary:         var(--lotusWhite3);
    --background-primary-alt:     var(--lotusWhite4);
    --background-secondary:       var(--lotusWhite1);
    --background-secondary-alt:   var(--lotusWhite2);
    --text-normal:                var(--lotusInk1);
    --text-faint:                 var(--lotusGray3);
    --text-title-h1:              var(--lotusRed);
    --text-title-h2:              var(--lotusOrange);
    --text-title-h3:              var(--lotusYellow);
    --text-title-h4:              var(--lotusGreen);
    --text-title-h5:              var(--lotusAqua);
    --text-title-h6:              var(--lotusBlue4);
    --text-link:                  var(--lotusBlue5);
    --text-a:                     var(--lotusTeal1);
    --text-a-hover:               var(--lotusRed);
    --text-a-url:                 var(--lotusTeal2); /*hover internal link, graph view link and standalone links*. External Links no*/
    --text-mark:                  rgba(183, 208, 174, 0.4); /* lotusGreen3 */
    --pre-code:                   var(--lotusGray);
    --text-highlight-bg:          var(--lotusBlue2);
    --interactive-accent:         var(--lotusBlue3); /* vim cursor background AND highilight on right*/
    --interactive-before:         var(--lotusGray2);
    --background-modifier-border: var(--lotusBlue3);
    --text-accent:                var(--lotusGreen);
    --interactive-accent-rgb:     var(--lotusGreen);
    --inline-code:                var(--lotusGreen);
    --code-block:                 var(--lotusGray3);
    --vim-cursor-backgroung:      var(--lotusGray);
    --vim-cursor-color:           var(--lotusInk1);
    --text-selection:             rgba(183, 208, 174, 0.5); /* lotusGreen3 */
    --text-tag:                   var(--lotusViolet1);
    --task-checkbox:              var(--lotusGray2);
    --table-header:               hsl(42, 33%, 91%);
    --table-row-even:             hsl(42, 35%, 94%);
    --table-row-odd:              hsl(42, 37%, 97%);
    --table-hover:                var(--lotusWhite1);
    --bold-color:                 var(--lotusInk1);
}

# Paper Version

Remixed Kanagawa color scheme with muted colors. For Neovim. See kanagawa-paper.nvim, my Obsidian Kanagawa Paper here, or others I created at Obsidian Community - sspaeti.

# History of the Great Wave by Kanagawa

Few paintings have lasted the test of time as the great wave of Kanagawa. It took Kanagawa 40 years before he drew the wave he is most known for. People that worked on the same idea for decades — Nitya’s Notes

Source: People that worked on the same idea for decades — Nitya’s Notes

Which is the source of Kanagawa Theme.

Origin: kanagawa

Obsidian Webclipper

Wed, 07 Oct 2026 19:13:51 +0000

Obsidian Webcliper is an alternative to ReadWise web clipper.

Super powerful to clip websites to my Obsidian vault. Even with added AI-interpreted (e.g. description, scanning a LinkedIn Profile, etc.) for a summary. Find all the features in the documentation.

# My Templates

You can download and import

Feel free to download my exported Webclipper setting and import it to yours with all the templates. Warning. Importing will overwrite all your existing settings; make sure you export yours first as a backup.

I created a template for my most used cases:

Default
Longer Summary
YouTube
Bluesky
Wikipedia Starter
Newsletter Summary
Books (Goodreads)
LinkedIn
Recipes Cooking
Coffee Beans
Hacker News

# Default

Here is an example of my default one

# {{title}}

Summary:
{{"a 5 bullet point summary, any titles, and references to people, places or books should be in [[wikilinks]] format"}}

---
Origin: {{url}}
Created [[{{date|date:"YYYY-MM-DD"}}]]

# Book (Goodreads)

note name: {{title|split:":" |first}} ({{author}})
note location: 💡 Resources/📚 Books/Want to Read/
template triggers: https://www.goodreads.com/

# {{title|split:":" |first}} ({{author}})

- Subtitle: {{schema:@Book:name|split:":" |last}}
- Link: {{url}}
- Tags / Categories: #📬 #📚 #reading #wanttoread
- Created: [[{{date|date:"YYYY-MM-DD"}}]]
- Author: [[{{author}}]]
{{"Extract published on, ISIN/ASIN and how many pages if you find the information and list them with separate lines with markdown listing `-`. Always call them `Published on: [[YYYY-MM-DD]]`, `ISIN or ASIN:` (depending on which one you find) and `Length: XXX pages`"}}
- Type: [[Books]]
- Genre:
- Related: 
- Started reading: 
- Finished reading: 
- Origin:

> 
> 
> TODO

## Short description (GoodRead)
{{"a short description of the book, maximum 400 characters"}}

## Notes During Reading
- ...

Rendered:

See more about the process of taking notes

How I Taking Notes for Physical Books

# YouTube

Interpreter context:

`1`	`{{transcript}}`

*I added a new lines here and below for readability, please remove them before

# {{title}}

{{description}}

## AI Highlights from [YouTube]({{url}}):
{{"summarize key points from this YouTube Video and transcript in bullet points. Use wikilinks format like [[wikiink]] for most important (keep the wikilinks to 3-4 max) titles, and references to people, places, or books"}}

## Transcript:
{{transcript}}

---
Origin:
Created [[{{date|date:"YYYY-MM-DD"}}]]

For the transcript to work, make sure you click Show transcript:

# Recipes Cooking

# {{title}}
{{description}}

![]({{image}})

## Ingredients
{{"List all ingredients needed for this recipe"}}

## Preparation
{{"Create a numbered list and copy all the steps to cook and prepare this dish, 
bolding the key parts (only 2-3 words) of each step in markdown format. Remove empty lines for listing"}}

## Inspiring
{{"Point out anything surprising or special about this menu"}}

---
Origin: {{url}}
Created [[{{date|date:"YYYY-MM-DD"}}]]

# LinkedIn

Interpreter context: {{content|strip_tags:("aside,iframe,ad-banner-container")}}

# {{title|replace:"/\s\|\sLinkedIn$/":""}}


## Background LinkedIn
*{{url}}*
{{"3 key bullet points highlighting their current role, industry focus, and unique value proposition. 
References to people, places or books should be in [[wikilinks]] format. Don't hallucinate."}}

{{" markdown table (comma instead of listings) with professional details like: Current Position, Company, Location, 
Industry, Education, Certifications, Projects (if there), Languages. Don't hallucinate."}}

---
Origin: {{url}} (LinkedIn)
Created [[{{date|date:"YYYY-MM-DD"}}]]

# AI Interpret

If you defined OpenAI of Claude, you can let AI prompt the AI like this:

1
2
3

## Summary

{{"a 5 bullet point summary, any titles, and references to people, places or books should be in [[wikilinks]] format"}}

# Specify HTML Tags

You can specify an HTML tag to analyze. E.g. for LinkedIn, as the full page was always too long for Claude:

`1`	`{{content\|strip_tags:("aside,iframe,ad-banner-container")}}`

# Examples

# Recipe

# GoodRead (Book)

# Coffee Beans

# LinkedIn Profile

# Variables

Variable	Description
`{{author}}`	Author of the page
`{{content}}`	Article content, highlights, or selection, in Markdown format
`{{contentHtml}}`	Article content, highlights, or selection, in HTML format
`{{date}}`	Current date, can be formatted using the `date` filter
`{{description}}`	Description or excerpt
`{{domain}}`	Domain
`{{favicon}}`	Favicon URL
`{{fullHtml}}`	Unprocessed HTML for the full page content
`{{highlights}}`	Highlights with text and timestamps
`{{image}}`	Social share image URL
`{{published}}`	Published date, can be formatted using the `date` filter
`{{site}}`	Site name or publisher
`{{title}}`	Title of the page
`{{time}}`	Current date and time
`{{url}}`	Current URL

Variables - Obsidian Help

# Further Reads

Are You Stuck in the Collector Trap? Full Obsidian Web Clipper Guide - YouTube

Origin:

Delta Lake

Wed, 07 Oct 2026 18:18:44 +0000

Delta Lake is an open source project that enables building a Lakehouse architecture on top of data lakes. Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing on top of existing data lakes, such as S3, ADLS, GCS, and HDFS.

Merging of APIs with the Acquisitions of Databricks over the Team of Iceberg

With the acquisition the APIs of Apache Iceberg and Delta will merge eventually. Mimoune Djouallah says that: “there will be two versions of delta, a classic one that write to a filesystem and the other that for all purpose is just a special implementation of iceberg.”
Hoyt Emerson says that “What I’ve heard is Delta features get pulled into Iceberg (i.e. deletion vectors).” See also more below in .

# Architecture and Storage

Databricks stores files in distributed object storage such as AWS S3, Google Cloud Storage or Azure Blob Storage in an open-source table format: Apache Parquet. Parquet is the de facto standard for storing column-oriented, compressed data. The encoded db-schema and the availability for multiple languages make it a great choice to store data for big data use cases.

The difference between the Delta Lake table format and a native Parquet file is the addition of the Transaction Log (Delta Lake). Furthermore, Delta Lake unifies different Parquet-Files with the same db-schema into a single folder, essentially the path to a Delta Table. Included in that folder is an automatically generated separate folder for the transaction log represented by _delta_log, which logs the ACID transactions happening on a table and allows neat features such as rolling back the data to a previous point in time.

# File Format Implementation

See Data Lake File Format.

# Understanding the Sync Process

Let’s look at what files Airbyte created behind the scenes on the S3 data storage, originally from Load Data into Delta Lake on Databricks Lakehouse.

I used CyberDuck to connect to S3, but you can connect via cmd-line or the web console. Below is an image that illustrates how that looks:

An illustration of a Databricks Lakehouse destination on AWS S3

Airbyte created several files. Most notably, the *.snappy.parquet and the _delta_log contain the above-mentioned delta lake transaction log. The content of your source data resides in the parquet files consolidated in a delta table represented as a single folder in the tutorial_cities and tutorial_users tables. Each table folder contains the delta-log with detailed information about each transaction.

There are other internal folders that Databricks created, such as copy_into_log and SSTs produced by the COPY INTO command.

The Airbyte Databricks Lakehouse connector does at its core two following steps:

CREATE OR REPLACE TABLE AS SELECT * FROM LIMIT 0

COPY INTO

These two transactions are also represented in the transaction log with 00000000000000000000.json/crs and 00000000000000000001.json/crc. If you open the transaction log represented by the two JSON and CRC files, you should see a protocol for each transaction that happened and some metadata.

00000000000000000000.json:

{
    "commitInfo": {
        "timestamp": 1654879959245,
        "operation": "CREATE TABLE AS SELECT",
        "operationParameters": {
            "isManaged": "false",
            "description": "Created from stream tutorial_cities",
            "partitionBy": "[]",
            "properties": "{"delta.autoOptimize.autoCompact":"true","delta.autoOptimize.optimizeWrite":"true","airbyte.destinationSyncMode":"append"}"
        },
        "isolationLevel": "WriteSerializable",
        "isBlindAppend": true,
        "operationMetrics": {
            "numFiles": "1",
            "numOutputBytes": "927",
            "numOutputRows": "0"
        }
    }

00000000000000000000.crc:

`1`	`{"tableSizeBytes":927,"numFiles":1,"numMetadata":1,"numProtocol":1,"numTransactions":0}`

And the second transaction 00000000000000000001.json with COPY INTO:

{
    "commitInfo": {
        "timestamp": 1654879970063,
        "operation": "COPY INTO",
        "operationParameters": {},
        "readVersion": 0,
        "isolationLevel": "WriteSerializable",
        "isBlindAppend": true,
        "operationMetrics": {
            "numFiles": "1",
            "numOutputBytes": "2085",
            "numOutputRows": "3"
        }
    }
}

00000000000000000001.crc:

`1`	`{"tableSizeBytes":3012,"numFiles":2,"numMetadata":1,"numProtocol":1,"numTransactions":1}`

# Key Features

General features are on Data Lake Table Format

# Schema Validation

Delta Lake automatically validates DataFrame schema compatibility with table schema during writes, following these rules:

All DataFrame columns must exist in the target table
DataFrame column data types must match target table types
Column names cannot differ only by case

Delta Lake on Databricks supports DDL for explicit column addition and automatic schema updates. When using append mode with partitionBy, Delta Lake validates matching configurations and throws errors for mismatches. Without partitionBy, it automatically appends to existing data partitioning.

Source: What is Delta Lake? | Databricks on AWS

# Schema Enforcement Implementation

Delta Lake employs schema validation on write, ensuring all new writes are schema-compatible before committing. Incompatible schemas trigger transaction cancellation and raise exceptions. The compatibility rules require that the DataFrame:

Cannot contain additional columns absent from the target table
Must have matching column data types
Cannot contain case-different column names

Source: Schema Evolution & Enforcement on Delta Lake - Databricks

# Change Data Feed (CDF)

Change Data Feed (CDF) feature allows Delta tables to track row-level changes between versions of a Delta table. When enabled on a Delta table, the runtime records “change events” for all the data written into the table. This includes the row data along with metadata indicating whether the specified row was inserted, deleted, or updated.

You can read the change events in batch queries using DataFrame APIs (that is, df.read) and in streaming queries using DataFrame APIs (that is, df.readStream).

# Manifest Files

For integration with Trino and similar tools, Delta Lake typically works with a manifest file. See Presto, Trino, and Athena to Delta Lake integration using manifests.

# DuckDB writes to Unity Catalog

TLDR: DuckDB’s Delta and Unity Catalog extensions shed their experimental tags — now with writes, Unity Catalog and Time Travel support:

more in this blog: Delta Grows Up: Writes, Unity Catalog and Time Travel | Delta Lake

# Use cases

Change Data Feed is not enabled by default. The following use cases should drive when you enable the change data feed.

Silver and Gold tables: Improve Delta performance by processing only row-level changes following initial MERGE, UPDATE, or DELETE operations to accelerate and simplify ETL and ELT operations.
Transmit changes: Send a change data feed to downstream systems such as Kafka or RDBMS that can use it to incrementally process in later stages of data pipelines.
Audit trail table: Capture the change data feed as a Delta table provides perpetual storage and efficient query capability to see all changes over time, including when deletes occur and what updates were made.

# Using DuckDB

duckrun runs SQL in DuckDB and reads/writes Delta Lake via delta-rs — locally or on OneLake / S3 / GCS / ADLS. It’s just glue: DuckDB executes · delta-rs materializes · Arrow bridges · dbt orchestrates. Two ways to use it:

connect() — a notebook helper to query and write Delta straight from SQL (this page);
a dbt adapter that materializes models as Delta tables.

# History

# Databricks Acquires Tabular

2024-06-04 Ali Ghods announced the acquisition of Tabular (Iceberg), the company behind Apache Iceberg. origin, See Data Engineering Acquisitions.

# Version Updates

Delta Lake 2.0
Delta Lake 3.0

# FAQ

# Vendor Independence

Delta Lake maintains independence through:

Linux Foundation management
Support across major cloud platforms (AWS, GCP, Azure)
Integration with various tools and services
XTable for format conversion
Delta Universal Format (UniForm) supporting Apache Iceberg and Apache Hudi interoperability

Delta is not only used by Databricks and there is no vendor locking as Delta Lake project is managed by the Linux Foundation. Delta Lake is supported on AWS Lake Formation, Athena, Redshift (even long time before the support for Iceberg was added there), and EMR. Snowflake does also have some support for Delta Lake. GCP BigQuery also does have some support for Delta Lake. Azure ADF, Synapse, also support Delta Lake. Microsoft Fabric is built on top of Delta Lake. Many other tools and services support Delta. So I don’t think you need Databricks or vendor locked there as you’re suggesting, if you go the Delta way. recap on Iceberg Summit 2024 conference: r/dataengineering

# Further Reads

Some references:

Building Analytics App on top of DeltaLake
Delta lake Hands-on
duckdb-deltatable-extension: A purely experimental DuckDB Deltalake extension

Origin: Data Lake / Lakehouse Guide: Powered by Data Lake Table Formats (Delta Lake, Iceberg, Hudi) | ssp.sh
References: Medallion Architecture
Created: 2019 when Spark & AI Summit Amsterdam 2019

Semantic Layer Measure Definition Examples

Wed, 07 Oct 2026 13:59:21 +0000

This list showcases different Semantic Layer definitions and examples.

# Example YAML Definitions

# Cube

Example from Cube with SQL Expressions:

cubes:
  - name: orders
    sql_table: orders
 
    measures:
      - name: statuses
        sql: "STRING_AGG(status)"
        type: string
 
    dimensions:
      - name: status
        sql: "UPPER(status)"
        type: string

Or more specifically designing Metrics:

views:
  - name: orders_view
 
    cubes:
      - join_path: orders
        includes:
          - status
          - created_at
 
          - completed_count
          - count
          - total_amount
          - average_order_value
 
      - join_path: orders.users
        prefix: true
        includes: 
          - city
          - age
          - gender

Example from Cube Docs

# Dbt Semantic Layer

semantic_models:
  - name: the_name_of_the_semantic_model ## Required
    description: same as always ## Optional
    model: ref('some_model') ## Required
    defaults: ## Required
      agg_time_dimension: dimension_name ## Required if the model contains measures
    entities: ## Required
      - see more information in entities
    measures: ## Optional
      - see more information in the measures section
    dimensions: ## Required
      - see more information in the dimensions section
    primary_entity: >-
      if the semantic model has no primary entity, then this property is required. #Optional if a primary entity exists, otherwise Required

Find all the properties and it’s use in Semantic models | dbt Developer Hub.

# SLayer

SLayer, a news OSS semantic layer.

# Models

Defines models this way:

name: orders
sql_table: public.orders
data_source: my_postgres
columns:
  - {name: id, type: number, primary_key: true}
  - {name: status, type: string}
  - {name: revenue, sql: amount, type: number}
measures:
  - {name: aov, formula: "sum(revenue) / count(*)"}

# Aggregations:

aggregations:
  - name: weighted_avg
    formula: "sum({value} * {weight}) / sum({weight})"
  - name: trimmed_mean
    formula: "avg(CASE WHEN {value} BETWEEN {low} AND {high} THEN {value} END)"

And support multi-stage metrics including Granularity:

  {
    "name": "monthly_store_revenue",
    "source_model": "orders",
    "measures": [{"formula": "sum(order_total)", "name": "rev"}],
    "dimensions": ["stores.name"],
    "time_dimensions": [{"dimension": "ordered_at", "granularity": "month"}]
  },
  {
    "source_model": "monthly_store_revenue",
    "measures": [{"formula": "avg(rev)"}],
    "dimensions": ["stores.name"]
  }
]

Examples from Models, built-in-aggregations, Multi-stage queries, made easy.

# Queries

Something other Semantic Layer do usually not provide, queries on runtime, specifies what data to retrieve from a model above.

Here a Dimension:

"dimensions": [
  "region",
  {"expression": "lower(city)", "name": "city_lc"},
  {"expression": "CASE WHEN amount > 2000 THEN 'big' ELSE 'small' END", "name": "size"}
]

Or grouping by an expression over an aggregate:

{
  "source_model": "orders",
  "dimensions": [
    "region",
    {"expression": "CASE WHEN sum(amount, partition_by=city) > 5000 THEN 1 ELSE 0 END", "name": "band"}
  ],
  "measures": [{"formula": "sum(amount)", "name": "band_total"}]
}

# YAML vs. SQL Comparison

dbt Semantic Layer, we can see tabs, use MetricFlow to define a semantic model that uses order_total as a metric and a sample schema to create consistent and accurate results which eliminating confusion, code duplication, and streamlining your workflow - example from dbt Developer Hub:

semantic_models:
  - name: orders    # The name of the semantic model
    description: |
      A model containing order data. The grain of the table is the order id.
    model: ref('orders') #The name of the dbt model and schema
    defaults:
      agg_time_dimension: metric_time
    entities: # Entities, which usually correspond to keys in the table. 
      - name: order_id
        type: primary
      - name: customer
        type: foreign
        expr: customer_id
    measures:   # Measures, which are the aggregations on the columns in the table.
      - name: order_total
        agg: sum
    dimensions: # Dimensions are either categorical or time. They add additional context to metrics and the typical querying pattern is Metric by Dimension.
      - name: metric_time
        expr: cast(ordered_at as date)
        type: time
        type_params:
          time_granularity: day
  - name: customers    # The name of the second semantic model
    description: >
      Customer dimension table. The grain of the table is one row per
        customer.
    model: ref('customers') #The name of the dbt model and schema
    defaults:
      agg_time_dimension: first_ordered_at
    entities: # Entities, which  usually correspond to keys in the table.
      - name: customer 
        type: primary
        expr: customer_id
    dimensions: # Dimensions are either categorical or time. They add additional context to metrics and the typical querying pattern is Metric by Dimension.
      - name: is_new_customer
        type: categorical
        expr: case when first_ordered_at is not null then true else false end
      - name: first_ordered_at
        type: time
        type_params:
          time_granularity: day

where as the SQL looks something like this

select
    date_trunc('day',orders.ordered_at) as day, 
    case when customers.first_ordered_at is not null then true else false end as is_new_customer,
    sum(orders.order_total) as order_total
from orders
left join customers
on orders.customer_id = customers.customer_id
group by 1, 2

# Open Standard

There’s now the Open Semantic Interchange (OSI) initiative, which resulted in Apache Ossie (renamed), which tried to interchange all semantic modeling with an **open-source, vendor-agnostic specification **.

Origin: Semantic Layer

SQL

Wed, 07 Oct 2026 09:40:44 +0000

SQL, a domain-specific language, is pivotal in programming, especially for managing data in relational database management systems or stream processing in relational data stream systems.

It’s closely tied to Jinja Template within dbt. Beyond Business Intelligence tools, a growing number of data engineering tools are now exclusively integrating with SQL. This is further explored in Building an Analytics API with GraphQL - The Next Level of Data Engineering.

For additional insights, refer to Analytics API[]

# SQL and Databases

SQL is the language of data. SQL is a fundamental skill for doing any data work. There’s almost nothing you do without needing SQL. If you work with a REST API with no direct SQL interface, it’s still beneficial to know, as the REST service will most certainly perform a SQL query against the database based on your REST request.

If you understand how joins and filters work, you can add a key filter to the request or split up a request into two, as you know it might be a heavy join behind, and that way you avoid it.

On the other hand, Python is a data engineer’s tooling language. It’s used to glue the different steps of a data engineer together. It’s the ultimate toolkit language. Pulling data from a REST API or web, cleaning out some insufficient data, and storing it in Postgres. How would you do that in a safe, ordered fashion? Right, Python.

In the old days, we used stored procedures. If you live in one ecosystem of large vendors like SAP, Oracle, or Microsoft, you can add your glue code into the database code with PL/SQL or T-SQL. But if you work with any outside tool, like a REST API, the tooling around Python is so massive and extensive that you will most certainly make your life easier by using Python.

# History

SQL was invented in the 1970s based on the Relational Data Model. It was initially known as the structured English query language (SEQUEL). The term was later shortened to SQL.

Oracle, formerly known as Relational Software, became the first vendor to offer a commercial SQL relational database management system.

“History of SQL”

SQL -> Data Mart -> Materialized View -> BI Report -> Traditional OLAP -> BI Dashboard -> Modern OLAP -> dbt tables -> One Big/Wide/Super Table -> Semantic Layer -> Natural Language Queries

# SQL vs. NoSQL or Big Data

Since then, SQL has come to dominate the landscape. (“Never bet against SQL,” I like to say.) Most users have moved to cloud SQL services (BigQuery, Snowflake) because convenience is more important than the flexibility and raw power of Apache Hadoop or Spark. SQL now handles workloads—streaming, data transformation, documents, geospatial, machine learning—previously handled by non-SQL systems. But there are two areas where SQL has not yet prevailed—Business Intelligence and data-intensive computing—and I want to investigate whether the solution is SQL or something else. Julian Hyde on Into the Wilderness

# ANSI SQL

What is ANSI SQL?

# SQL Syntax

Nice new features: UNION ALL BY NAME

# Modern SQL

There’s also a Modern SQL: A lot has changed since SQL-92 website by Markus Winand with showcasing the changes of SQL.

# Declarative

SQL is declarative.

# How it Works

# Different Parts of an SQL-Statement

From parsing to compiling and executing. Possible with SDF and SQLGlot:

source

# Joins

Temporal Joins
RewriteY: Rewrite multi-join SQL into faster plans. Analyze query shape and workload overlap to find reusable intermediates.

# The Future of SQL

# Extending SQL for Analytics

Julian Hyde in 2025-06-04 at data council More Than Query Future Directions of Query Languages, from SQL to Morel - YouTube:

SQL is the origin of data, and it has been extended over the years. We have geospatial capabilities, window functions, and other features throughout the year. Why not add analytics semantics, too?

More at Extending SQL for analytics.

# Patterns of SQL

From Matthias Broecheler on LinkedIn:

Everything gets pulled back in.

MapReduce? Became Hive. Then SparkSQL.

NoSQL? Morphed to “Not Only SQL” and then NewSQL.

Graph? Led to pattern matching inside SQL.

Vectors? SQL index type

Streaming? Enter: Flink SQL.
He shares more on The Gravitational Pull of SQL - by Matthias Broecheler.

# Resources

Recommended Reading: Efficient Data Processing in SQL by Joseph Machado
Comprehensive Guide: Ultimate SQL Guide from Count-co
Learning SQL as a detective: SQL Noir

# Further Reads

Query Planner

# Just for Fun

Chess in Pure SQL - DB Pro Blog

Origin:

Google Docs: New Native Markdown Integration

Tue, 06 Oct 2026 21:48:15 +0000

Google Docs is the king of collaboration, and the online version of Microsoft Word.

Open an new Docs Quick?

Did you know that you can open a new Google Docs document with docs.new? Try it 😉

# Google Docs Can Handle Markdown, Finally

As of 2026-10-06, Google announced to work with Markdown, natively:

We are making Markdown files work natively across Google Drive and Docs.

You can now preview .md files in Drive, open them in Docs, edit them, comment on them, and collaborate without converting the file into a Google Doc!

# Workflow before Native Markdown Support

Before you had to enable a specific configuration and paste and copy as Markdown:

Convert Markdown to Rich Text (Google Docs)
Wow, I just learned even another, easier way: Copy your MD to the clipboard and Paste from Markdown, I didn’t know they added this feature. Bravo Google Docs team 👏🏻.
Please Activate
This option is not activated by default. The “Copy as Markdown” and “Paste from “Markdown” options will be OFF by default and can be enabled in Docs by going to Tools > Preferences > Enable Markdown. Visit the Help Center to learn more about using Markdown in Google Docs, Slides, & Drawings.

And then you can simply Paste from Markdown (and Copy as Markdown)

Even images work !!

Another option to export can be achieved with Copy as Markdown, or download Gdoc file as MD file with File -> Download -> Markdown (.md).

# Export MD directly with Google Docs

TIL (2024-10-14), you can import a Markdown file directly with Google docs File -> Open or Open with Google Docs from Drive. Read more on Google Workspace Updates: Import and export Markdown in Google Docs.

# Google Docs Alternatives

Origin: Convert Markdown to Rich Text (Google Docs)

Markdown

Tue, 06 Oct 2026 18:54:42 +0000

A declarative way of writing text.

As written by Derek Write plain text files and summarized them in my
Personal Knowledge Management Workflow for a Deeper Life.

Markdown has excellent features, such as Admonition (Call-outs)s.

Markdown is the bridge between simplicity and power, while Vim’s modal editing is the master key unlocking efficiency. Together, they elevate plain text into an automated, format-rich masterpiece, free from platform lock-ins. Your words deserve this freedom.
(on another note: Neovim)

Explained in How Markdown took over the world - Anil Dash, it’s a way of making Markup and its Markup Language simpler:

If mark_up_ is complicated, then the opposite of that complexity must be… markd_own_. This kind of solution, where it’s so smart it seems obvious in hindsight, is key to Markdown’s success.

# Why Markdown?

Why Markdown? It’s just a Plain Text File. And if computers are still here in 100 years, Markdown will still be readable with every app that can handle text files.

And it just so happened that AI Agents with skills love it too, but the key is for you, the longevity of your thoughts and ideas.

# History

Creator is John Gruber.

# Tools / Editors

Tools to write Obsidian-like graph networks open-sourced: Open-Source Obsidian Publish Alternatives

# Static Site Generators

See in Static Site Generators (SSG) such as GoHugo.

# Closed Source:

Obsidian
- Example of public Data Engineering Wiki with Obsidian: Data Engineering Wiki - Reddit
Roam
Simple: Typora

# Collaborative Markdown tools:

HackMD: Efficent Markdown collaboration
- Add external folders (git-blog-book) to my Obsidian Vault via Symlink
Stashpad - Your streamlined Google Docs alternative
- check at Stashpad - Your streamlined Google Docs Alternative
Twitter to Markdown: the.rip
Markdown Drawing
DXOS?

# Simple Texteditors

# What’s the Difference between Markdown and Rich Text?

See Markdown vs Rich Text.

# How to convert Markdown to Rich Text (GDocs) and vise versa?

Convert Markdown to Rich Text (Google Docs)

# Market

# 2024

Obsidian and Markdown Plain Text Files are the most loved tools for developers in 2024.

Markdown wins the popularity contest with developers this year! Not only do users say they admire it more this year than last (84% vs. 82%), but Markdown also received the highest admired score of all technologies on our survey this year.

Source: Technology | 2024 Stack Overflow Developer Survey

^4f88aa

# 2023

Markdown files are the second most desired asynchronous tool and the most admired asynchronous tool. MD files can be deployed in various hosted instances and show an opportunity for paid solutions to reduce friction for sharing information. Stack Overflow Developer Survey 2023

Interesting, next admired are GitHub Discussions, also Markdown. Loving that.

# Limitations of Markdown

Some use Astro like Maggie Appleton to vizualize in her blog, e.g. like Growing a Human: The First 30 Weeks. And that’s certainly a limit of Markdown.

But for my writing and My Websites, I want to stay in Markdown as much as possible. Charts with Mermaid and tables I use everyday all day long. It also adds friction to not spend to much time on visuals, tough they can make all the difference, but my focus is on the words.

I am afraid of using new formats MDX and JSX, tough for testing it would be good. But so far I didn’t had a reason to change, because I couldn’t do anything. This is why I keep using GoHugo where I can write plain Markdown, seperated from the visuals or website design.

# Formats: Open Knowledge Format (OKF)

Open Knowledge Format (OKF)- Typed Markdown

# Further Reads

Origin:
References: GoHugo

Neomd

Tue, 06 Oct 2026 08:31:12 +0000

Neomd is a keyboard-first TUI email client where you write in Neovim, render as Markdown, screen senders first, organize emails once. Neomd is my way of implementing a email TUI based on my experience on Neomutt focus on Neovim (input) and reading/writing (Markdwon) and navigating, Vim Motions.

Keyboard-first TUI email: write in Neovim, render as Markdown, screen senders first, organize emails once. https://t.co/ACVNQe8X0K pic.twitter.com/893Iua4NjN
— Simon Späti 🏔️ (@sspaeti) May 1, 2026

It’s a terminal email client that lets you compose and read emails in Markdown using your $EDITOR (neovim), rendering them as minimal but beautiful HTML on send with possibility of all Markdown features and Callouts — all from a single bubbletea TUI. It features a HEY-inspired Screener workflow where new senders are triaged into accept/reject lists before hitting your inbox, turning email into a GTD-friendly, and triage emails only once workflow.

This is how it looks when reading the Feed (dedicated place for email newsletters):

See almost all feature: reading newsletter directly in your email client (feed) - see spy pixel, if you have replied (dot) and thread mode if replied)

Reading view on the right, different screener list below:

Find a short rundown on YouTube:

# What is Unique about Neomd?

The key here is speed in which you can navigate, read, and process your email. Everything is just a shortcut away, and instantly (ms not seconds). It’s similar to the foundations that Superhuman was built on: it runs on Gmail and makes it fast with vim commands.

With the HEY-Screener, you get only emails in your inbox that you screened in, no spam or sales pitch before you added them. Or don’t like them, just screen them out, and they get automatically moved to the “ScreenedOut” folder.

With the GTD approach, using folders such as next (inbox), waiting, someday, scheduled, or archive, you can move them with one shortcut. This allows you quickly to move emails you need to wait for, or deal with later, in the right category. Processing your email only once.

With the additional Feed and Papertrail, two additional features from HEY, you can read newsletters (just hit F) on them automatically in their separate tab, or move all your receipts into the Papertrail. Once you mark them as feed or papertrail, they will moved there automatically going forward. So you decide whether to read emails or news by jumping to different tabs.

# Core Features

Only a few of the features:

Markdown compose — write in $EDITOR (nvim default), send as multipart HTML + plain text
Attachments — via yazi or <leader>a in nvim; images inline, 1–9 to download
Preview emails including images.
Open HTML emails such as Substack or others with w3m
Multiple from addresses — SMTP-only aliases, cycle with ctrl+f
Glamour reading + Kanagawa theme — emails as styled Markdown, nvim palette
Markdown formatting with bold, links, and italic. Formatted in-line and code blocks, header h2, h3
Clever and sophisticated shortcuts to move or screen emails.

Find all at GitHub - ssp-data/neomd: A minimal email TUI where you read with Markdown and write in Neovim. · GitHub.

# How to Use

Read the details with all its shortcuts at GitHub.

flowchart TD
    Start([New Email Arrives]) --> AutoScreen{Auto-Screener
Known Sender?}

    AutoScreen -->|screened_in.txt| Inbox["📥 Inbox (Next)"]
    AutoScreen -->|screened_out.txt| ScreenedOut[🚫 ScreenedOut]
    AutoScreen -->|feed.txt| Feed[📰 Feed]
    AutoScreen -->|papertrail.txt| PaperTrail[🧾 PaperTrail]
    AutoScreen -->|Unknown| ToScreen[❓ ToScreen]

    ToScreen -->|Press I| ClassifyIn[Add to screened_in.txt]
    ToScreen -->|Press O| ClassifyOut[Add to screened_out.txt]
    ToScreen -->|Press F| ClassifyFeed[Add to feed.txt]
    ToScreen -->|Press P| ClassifyPaper[Add to papertrail.txt]

    ClassifyIn --> Inbox
    ClassifyOut --> ScreenedOut
    ClassifyFeed --> Feed
    ClassifyPaper --> PaperTrail

    Inbox --> Process{Process Email
GTD Decision}

    Process -->|< 2 min?
Do it now| Action[Reply/Handle
Immediately]
    Process -->|Waiting for others
Press Mw| Waiting[⏳ Waiting]
    Process -->|Not now, later
Press Mm| Someday[📅 Someday]
    Process -->|Time-specific
Press Mc| Scheduled[🗓️ Scheduled]
    Process -->|Delete
Press x| Trash[🗑️ Trash]
    Process -->|Reference only
Press Mp or P| PaperTrail
    Process -->|Newsletter
Press F or Mf| Feed

    Action --> Done{Done?}
    Waiting --> Review[Review Later]
    Someday --> Review
    Scheduled --> Review
    Feed --> ReadLater[Read in
Feed Tab]
    PaperTrail --> SearchLater[Search when
needed]

    Done -->|Yes| Archive[📦 Archive]
    Done -->|Not actionable| Archive
    Review --> Archive
    ReadLater --> Archive

    classDef folderStyle fill:#54546d,stroke:#7fb4ca,stroke-width:2px,color:#dcd7ba
    class ToScreen,Inbox,ScreenedOut,Feed,PaperTrail,Archive,Waiting,Someday,Scheduled,Trash folderStyle

all colored boxes represent neomd folders

Key principles:

Screener first: Unknown senders never clutter your Inbox — they wait in ToScreen for classification
One-time decision: Once you classify a sender (I/O/F/P), all future emails from them are automatically routed
GTD processing: Emails in Inbox are processed once — if < 2 min, do it or keep it in inbox as doing Next otherwise move to Waiting, Someday, or Scheduled
Minimal filing: Only Archive when done; no complex folder hierarchies — use search to find old emails
Separate contexts: Feed for newsletters (read when you want), PaperTrail for receipts (search when needed)

# History

Find the full history at Twitter - inspired by seeing Email.md on HackerNews.

It’s fully Vibe Coded, I only omitted the taste with pointing it to all my tools and astetics, also showing it the code from open source tools.

# Inspiration

See inspirations for this project such as:

Neomutt — the gold standard terminal email client; neomd reuses its screener list format and borrows many keybindings
HEY — the Screener concept: unknown senders wait for a decision before reaching your inbox
hey-cli — a Go CLI for HEY; provided the bubbletea patterns used here
HEY-Screener in (Neo)Mutt: My way of using HEY screener in neomutt.
Newsboat — RSS reader whose O open-in-browser binding and vim navigation feel inspired neomd’s reader view
emailmd.dev — the idea that email should be written in Markdown
charmbracelet/pop — minimal Go email sender from Charm
charmbracelet/glamour — Markdown rendering in the terminal
Kanagawa — the color palette used for the inbox
msgvault — Go IMAP archiver; the IMAP client code in neomd is adapted from it

If you rather see the prompt, check out initial prompt and its plan.

# Integrations

# It Works Suprisingly Well on a Mobile Phone

Not that I would use it all day long, but to browse my newsletter emails, or checking the emails, it’s really nice on a phone with Terminus app:

The Screener is running on my home-server

So that emails get automatically screened out and moved to ToScreen folder, I run Neomd as a Headless Daemon Mode on my homeserver - syncing emails every 2 minutes.

# Omarchy Plugin

Now also as Omarchy plugin womarchy-neomd-plugin to browse the emails right there from the top bar.

Full video at This Tweet.

# Roadmap

Some ideas in random order:

Themes:
- 1. make them configurable with extra config
- 2. make an omarchy theme, that uses omarchy color paletter from obsidian or neovim or Omarchy itself, so when we change omarchy theme, neomd will also change.
12h option next to current 24h option in reply emails and list items
reminders for email if not replied yet only.
takes a while or hangs sometimes on Wifi or internet connection change
light mode shoudl have different text color (optimize for light mode or toggle automatically when light background)
When I reading an email: shortcut for listing all emails from that domain (and previous emails - similar to thread view, but based on domain, email instead of subject name)
Sent later feature: https://neomd.ssp.sh/docs/sending/#send-later
contacts.csv from google read only integration https://neomd.ssp.sh/docs/configuration/#contacts
store lists on IMAP?
- https://www.manpagez.com/man/1/alpine/ - similar to what Alpine/Neomutt is doing? (see Issue on github)
- optional - local first and offline should still work - sync between them?
emoji reaction - should we add a json attachment, gmail does that like this: { "emoji": "👍", "version": 1 } with the name attachment.vnd.google.email-reaction+json
extend serach of emails with DuckDB - see Full-Text Search with DuckDB - peterdohertys.website
- also check msgvault
people can add their own folder: make it dynamically? (currently hard coded) - see FAQ – neomd Documentation
offiline mode (?): this is a workflow change and atm not priority, but downloading offline to batch reply on an airplane would be great. Would need encryption too, that we download all emails encrypted. Potentially encrypting our lists too? download with POP3
- T4T5 on X: “@sspaeti the thing i really want is scriptability i’m thinking that if that if my data is stored as simple plaintext files, i can easily ask agents to write useful workflow scripts i’d like to scan my emails for calendar events to put in caldir, receipts to store in beancount, etc” / X

# Images in Neomd

image rendering with neovim Image API: https://github.com/neovim/neovim/issues/30889 (would be only during composing, not reading)
- not inside goldmark, as this does not work. Matcha uses sixel + kitty + iTerm2 + ghostty via terminal env-var detection, renders out-of-band by writing to stdout with cursor positioning (bypasses bubbletea). Uses mattn/go-sixel + stdlib image/png. and does not use glamour: has its own HTML→element pipeline (view/html.go, 1087 LOC).
Inline image rendering · Issue #20 · ssp-data/neomd
- 👀 goldmark-images: Goldmark image replacer extension · GitHub
! quickshell (e.g. https://github.com/sspaeti/omarchy-neomd-plugin) CAN SHOW images! maybe we could use that? to be checked out

# DONE

signature per-account
- (see Issue on github)
check similarity with matcha TUIs
Get notifications for certain domains/senders on Hyperland (integrated in notifications)
approving full domains, not individual senders only
Emoji reaction should be sent from the account the email was sent too
I use Listmonk for my email, it’s open source too, so I’m thinking of integrating an email or an integration that let’s me send directly from neomd to all my subscribers.. 🤔
! deamon to run on NAS or server, to automatically run screener to have screened emails also when opening on mobile
- Background sync daemon — separate neomd-sync binary (or cron job) that runs IMAP fetch + screener headlessly, so the inbox stays clean even when neomd is closed; complement to the in-process 5-min tea.Tick sync already running while neomd is open
- ? Should we use syncthing to sync text-files between my desktop, server and android phone?
(I just use normal spellchecker, works well)
- integrate Aspell (or any other spellchecker) into neomd - here someone integrated it into neomutt: Still loving the Micro Journal Rev.2 with Neovim—Chill writing session. : r/writerDeck
- has normal spell checker from neovim. not working super nicely yet, but it works: dotfiles/nvim/.config/nvim/lua/sspaeti/set.lua at a66451e7f29afce97f44d0965d31c0fa39d29c61 · sspaeti/dotfiles · GitHub
set up IMAP with proton.me and mail bridge https://proton.me/mail/bridge
Add callout support
- https://github.com/VojtaStruhar/goldmark-obsidian-callout
- https://gitlab.com/staticnoise/goldmark-callout
💡 Feature: Emoji reaction reply · Issue #9 · ssp-data/neomd
Showing and being able to attach files
Change A for switching account not a
cmd mode like vim wtih :
fix search? (e.g. find all keys that are shortcuts like above a, or disable shortcuts when searching?)
integrate Newsboat into Feed of my email? -> not great idea
Unread counts in folder tabs — e.g. Inbox (3)
having a signature when composing an email
d / u half-page scroll in reader (vim-style)
Reply-all — group reply to all original recipients
CC field in compose
Forward email — f key, pre-fills “Fwd: " subject with quoted body
Add an Attachment, quite important: the flow should be easy from within Neovim as this is the composer
choose sender in composer: setup gmail, hey, and others in one account to send - without need to switch account
highlight threads or conversations that are forth and back in a thread
Docs instad of readme.md?
better search: overall and keeping search when to apply multi command on search result
integration with contact or email - reading from screened_in list?
create a demo account for showcasing the features (e.g. neomd.demo@gmail.com)j
- create a hostpoint demo account, as gmail is way slower… defeates neomd
- when finished or restart
  - ./scripts/reset-demo.sh welcome screen
  - add or create truncates the screener lists (doesn’t delete them) and removes the welcome marker.
always on bcc in config.toml (if each email should be sent to your other email for example)
### Not priority right now
Background sync daemon — separate neomd-sync binary (or cron job) that runs IMAP fetch + screener headlessly, so the inbox stays clean even when neomd is closed; complement to the in-process 5-min tea.Tick sync already running while neomd is open
looks pretty nice, looking forward to that keychain support merge, or if you supported command line argument to fetch password like neomutt that would also be great PASS=pass xyz Reddit suggestions

Full Change log at neomd/CHANGELOG.md at main · ssp-data/neomd · GitHub.

# Presenting and Key Features

Points to showcase, see the video here - Neomd: Vim-native Email TUI with Markdown Compose & HEY Screener | Take 2 - YouTube:

opening new account
- welcome screener
- using neovim, yazi
- kanagawa theme
email start in inbox
- amount of loaded emails
screener
- and screen-all if you want
- GTD
- tabs
move around, move emails
- speed
- lists
- sorting
! search - search overall
reading emails: markdown
- opening w3m, browser AND nvim
- images
composing
- markdown native
- images attach
- send forth and back in neomd
- signature
- Preview
- drafts
- different [senders] and [accounts]
- autocomplete emails
saving drafts - UNDO
threaded view and reply DOT
- threaded view with T
comparision neomd with IMAP web view -> Always in sync if accessed from mobile
! everything folder
zoom out and jump to next read email
keyring
theming
ai integration, as it’s just a local markdown file
(Calendar integration)

# Further Reads

Email and Email TUIs
Email, and the way we (should) communicate at work | ssp.sh
Learning from someone else’s vibe coded project: Im going back to writing code by hand | k10s devlog
Gmail Thinks I’m Stupid, So I Left | moddedbear.com

# Reading RSS Feed vs. Reading Newsletters in Email

Reading in Neomd (Email) vs. Newsboat (RSS):

Simon Späti 🏔️ on X: “Reading in Neomd (Email) vs. Newsboat (RSS). https://t.co/ViIEfEFhIF” / X

# A Happy Neomd User

Just for fun, a happy neomd user 😉.

Generated with AI Meme Generator - Create Viral Memes in Seconds | Glif

Origin: Vibe coded after seeing Show HN: Email.md – Markdown to responsive, email-safe HTML | Hacker News

Taxonomy of My Note Types

Mon, 05 Oct 2026 20:45:08 +0000

Taxonomy I started with, and what I updated to. This is basically how my notes grow on my second brain from: started -> growing -> evergreen. Usually they start with an idea, quick capture, some rough notes, usually just a Fleeting Notes in my private Obsidian vault.

If they get more edits, or I think they are non-sensitive and valuable for others immediately, I just check the grammar, write some more paragraphs or sentences, and directly publish them. These are the notes tagged with #started. When they have enough changes, or are up for a long time, they get an update to #growing. These notes are ideas or writing I have grasped for a longer time, and I wrote substantially more; then they get the growing tag. If it’s a fully written article or essay, and I have substantially written and refined it, they go into #evergreen notes. These are the latest stage. That does not mean they get no more updated, but that means you can see them as well crafted and in a state to be shared, discussed, etc. (not that the others are not).

Evergreen ≠ finished.

All notes are alive, and continue to compound, these are Continuous Notes eventough categoriez as Evergreen note. Evergreen means the note is well grown and matured, but still growing.

Also, nothing as part of my Second Brain is ever finished, that’s the whole idea. Finished thoughts or none changing are published as blogs, as blog vs. notes explains further. ^6560b7

# The Goal: Giving Indication to You as the Reader

But the main idea is for you as the reader to give an indication of how big my investment was; you also see the recent commits via dates and words removed/added. So you get a good sense of it.

And for me as the writer, to have an easy way to share preliminary but valuable ideas early on.

# The Tagging Mechanism

So that the statuses are always up-to-date and accurate, and I don’t need to spend too much time on them, there’s an automatic process in place based on git commits and words and dates - plus I have the option to overwrite it manually going forward with a setting the status myself if I want to make sure that one is not yet evergreen eventough I worked a lot on it or it’s very long, or vise-vera if it’s short but I spend many hours refining it.

The rules currently work like this (first match wins):

Manual tag #garden/<level> set by me (will be shown in the pop-up)
Evergreen: ALL three: words ≥ 600 AND sessions ≥ 5 AND span_days ≥ 180.
Started: words < 50 (tiny, regardless of history) OR (words < 120 AND span_days < 30) OR fresh: created in my vault within the last 30 days AND words < 300 (a note I just wrote is a quick capture however long, it will grow into the next level by itself after a month).
Growing: everything else.

# How My Internal Tags and Public Tags Work Together

So I map my old ones to new ones like this:

Level	Reader-facing sentence (popover)	Vault mapping
`started`	Started. Quick capture, barely worked on. May change drastically.	`📬`, fresh short notes
`growing`	Growing. Worked on, still rough. Expect bullets, gaps and change over time.	`🗃/🌻`, `🗃/📖`
`evergreen`	Evergreen. Own words, refined over time. Still grows.	`🗃/🌳`

See also SecondBrain Disclaimer. These notes have been updated 2026-10-01 as old Taxonomy was not active and is no longer used.

Inspiration from Maggie and Andy

Inspired by Maggie Appleton on her Digital Garden and the talk where she shared a bit about the background story. She also helps define the Digital Garden concept (see her Visuals), and she uses Seed, Budding and Evergreen for her stages, similar to Andy Matuschak.

# Metadata and Frontmatter

Additional Frontmatter and metadata such as:

aliases: different names for the same note
Origin: Where the idea/thought came from, sometimes also the parent.
Up: sometimes added to showcase parent, as Origin is usually the idea (currently Origin is both)
children:? Thought about, but not really see the use yet
map or 🗺 : Map of Content (MOC) - bundle multiple notes as one.

Origin: Organizing and Writing My Notes
References: Taxonomy of Note Types - before

Plain Text Files

Mon, 05 Oct 2026 20:43:30 +0000

Knowhow stored in Notions and Google Docs will not scale (what if you need to change something?), we can’t do a search and replace. But why not use what developers have for ages for knowledge, too?

Plain text files are future-proof:

Every device, including ones long gone, and ones not invented yet, can read and edit plain text. Whether future virtual reality or a chip you can implant in your earlobe, plain text will be there. Will Microsoft Word? Evernote? Notion? Maybe. Maybe not.

Plain text files are:

Portable
Un-Commercial
Offline
No Dependencies
Easiest to convert

# Markdown

This is the story of how Neovim/vim and Markdown unified my data engineering and writing workflow. As Kepano, the CEO of Obsidian says:

Obsidian will not exist forever, no app will. However, the files you create in Obsidian are yours, and can hopefully last for generations. File Over App

For me, Plain text files are the foundation, Markdown is the best plain text format there is, it’s is a text file still, but you have some formatting that are stored within the text, and Neovim and any other editor can open it in 100 years still. They will last until the end of days of computers. Read more at Markdown note.

# Relates

# Local First

Local First is an approach that heavily uses plain text files, and what Markdown and Obsidian for example use.

# Markup Languages

See more at Markup Language.

# Claude Skills

Arent’ Agentic Skills just Plain text files?

# Further Reads

Markdown vs Rich Text.
File Over App
Creating Local First Collaboration Software with Automerge - YouTube by Daniel Kahneman
Here’s a good video on how to do that and work with a team, mainly with git and GitHub, but all with Plain text: The Plain-Text Team.
Text-based web browsers - CSSence.com
Why Plain Text Is Still One of the Best Technologies We Have | Dead Parrot BBS

Origin:

Terminals

Mon, 05 Oct 2026 16:23:52 +0000

Terminals are so powerful, especially with TUIs and Neovim. I love the terminal and the workflows (writing, data engineering, note-taking) it provides.

# Example of Querying a Database in the Terminal with Nvim

Here an example of a advanced workflow:

See the numbers explained and what is done at Query databases in vim (vim-dadbod)

# Terminal Apps

Below some tools and related stuff to terminals.

# Usuals

iTerm 2
PuTTY (Windows)
Windows Terminal

# GPU Enhanced

Kitty
Alacritty
WezTerm

See Kitty vs Alacritty

# Sophisticated AI/Collaborative

Warp
sshx

# Open-Source

Terminal7
Ghostty
Foot: CPU focused

# MacOS

cmux: Native macOS app built on Ghostty. Vertical tabs, notification rings when agents need attention, split panes, and a socket API for automation.

# On Libghostty

Terminals built on libghostty, the backend of Ghostty:

Ghosthub by Wes McKinney: A native power terminal for your tmux fleet

# Wiki List

List of terminal emulators - Wikipedia

# Which Terminal to use?

I don’t mind so much about which terminal, as long as it’s inside the terminal. What you see is above is Tmux. I currently use Foot (minimal), but I love Kitty, too. Herdr, another Terminal Multiplexer, might be good if you have 10s of Agents running. I’m a modest guy, I have no more than 2-3 running, so I don’t need Herdr at the moment, and Tmux is enough for me (with this agent integration tmux).

Try a bunch, and see if it works for you. That’s always the best. I like a distraction-free environment, and I feel that Herdr is too focused on agents, which is not what I need. But I’m also not coding too much, so that might change if I built lots of stuff.

# Futher Reads

tmux and other Terminal Multiplexer.
My Terminal Workflow with MacOS and Linux

Origin:

Omapic

Mon, 05 Oct 2026 13:52:04 +0000

Omapic, is a small little tool to Horizontal and Vertical Cut Out like Snagit does on Hyprland and Omarchy, heavily inspired by omacut.

I have ported the feature into Omasnap

See PR, please use Omasnap instead.

See the full video at GitHub. Full installation guide and features at README.md.

Disclaimer

This tool was fully vibe coded with the inspiration of Omacut. Simlar to Neomd and Pollmd, two other tools I build soley for me, but sharing online.

# Roadmap

Adding features of snagit, such as being able to add annotations, but keep moving after it was placed

Origin: Horizontal and Vertical Cut Out

Every Database Migration is One Too Many

Mon, 05 Oct 2026 13:47:40 +0000

How many times have I had to migrate from Microsoft SQL to Oracle or to any other database, or switch the Cloud Data Warehouse or Cloud Providers, sometimes for reasons, and sometimes just because management decided so.

Mostly, one Data Migration is one too many. This also counts for Open Data Stacks, or for data pipelines from one framework to the other or to change language. But on the other hand, these are real-life events we can’t fully avoid, especially if tools and companies get acquired.

I’d say one more database migration, or tool migration, is one too many 🙂. There’s a lot of SQL Transpiler that can convert from one dialect to the other. There’s even a transpiler for semantic layer specifications conversion with Apache Ossie. However, the time to do so, and to make sure every edge case is still working as part of the ETL pipeline or in the dashboard, is not as simple as “LLM migrate”, unfortunately, but for sure it helps.

# Read More

Origin: Data Migration, later: Discussio on LinkedIn
References: Excel Never Dies

The ssp.sh Newsletter

Mon, 05 Oct 2026 13:35:00 +0000

ssp.sh

Learn for Life

Data engineering, writing & a public Second Brain, all written by hand, to your inbox.

Free for you · only sent when I have something worth sharing.

You'll get a confirmation email, click the link to finish signup.

Simon Späti
data engineer, technical writer & author. Enoying my craft.

Editions are email-only, but see the backlinks for a few older ones that I uploaded, and subscribe to get all of them.

It’s a classic newsletter like we had before Substack was a thing. I don’t send blogs by email (anymore), it’s me writing to you with what’s on my mind and sharing the latest news from my page and what’s going on. See it as random rambles that you don’t find anywhere else, not uploaded, email only.

I try to share interesting things to hopefully give you ideas or things that are unexpected, sometimes nerdy computer stuff, but also life, philosophy, and other things I am interested in at the moment, when learning every day.

Mostly thoughts about my experiences from my work in data engineering, technical writing, and my life as a dad living in Switzerland. Some behind-the-scenes from my Obsidian and Neovim workflow, what I’m learning, and news before it hits socials. I’m sending now and then, no schedule usually, but currently trying a monthly cadence.

Prefer to subscribe another way?

Subscribe on the dedicated page, or via curl sub.ssp.sh/your@email.com (or run curl sub.ssp.sh/why to see why to follow).

# Cadence and Origin

This blog started its first editions in 25th of July 2017 with Mailchimp, sent to a massive 5 subscribers 😉. Since then, I have sent 78 total emails, most of which were announcing a new blog post.

Fully handwritten, since I’m doing this when I switched to Listmonk in April 2023, I’ve sent about 25 newsletters that way, over roughly three and a half years, that’s about one every seven weeks. That should give a good overview of the cadence of this newsletter.

Some nostalgia, from the first edition sent to the five email subscribers (probably all demo accounts of mine 😉), to how it grew over the years and how I switched from Mailchimp to Listmonk.

This is what the first edition looked like, I can see, even then, that I wrote rather long 😅:

If you want to read that, this is mostly a re-send of my blog article, still live at What Data Warehouse Automation tools are on the market.

Today, the newsletter reaches about 3,200 readers per edition ( 4,2K sign-ups in total, minus unsubscribes, bounces, and unconfirmed addresses).

xychart-beta
    title "Newsletter Readers Reached by Year"
    x-axis [2017, 2018, 2019, 2020, 2021, 2022, 2023, 2024, 2025, 2026]
    y-axis "Subscribers" 0 --> 3500
    bar [6, 25, 32, 58, 100, 157, 470, 1777, 2842, 3219]
    line [6, 25, 32, 58, 100, 157, 470, 1777, 2842, 3219]

Looking forward to having you as part of the journey. Feel free to reply to any of the issues, the replies always go to my inbox directly.

Origin: subscribe.ssp.sh

Entity Relationship Diagram (ERD)

Mon, 05 Oct 2026 10:56:53 +0000

An Entity Relationship (ER) Diagram is a type of flowchart that illustrates how “Entities” (usually tables within a database), such as people, objects, or concepts, relate to each other within a system.

ER Diagrams are most often used to design or debug Relational Databases in the fields of software engineering, business information systems, education, and research.

Also known as ERDs or ER Models, they use a defined set of symbols, such as rectangles, diamonds, ovals, and connecting lines, to depict the interconnectedness of entities, relationships, and their attributes. They mirror grammatical structure, with entities as nouns and relationships as verbs.

Source Wikipedia.

# ERD Tools

Data Modeling Tools

# Further Reads

My three part series on Data Modeling and The Unsung Hero of Data Engineering:

Part 1: An Introduction to Data Modeling | ssp.sh
Part 2: Modeling Approaches and Techniques | ssp.sh
Part 3: Architecture Pattern, Tools and the Future | ssp.sh

Others:

Data Modeling Frameworks
Data flow diagram (DFD)
Semantic Models

Origin: Data Modeling – The Unsung Hero of Data Engineering- An Introduction to Data Modeling (Part 1)
References: More examples at Simon/Sync/3 Resources/Templates/Business Intelligence/ERD

Evergreen Notes

Mon, 05 Oct 2026 10:15:36 +0000

Evergreen notes are notes I have I have fully written and optimized over a long time range, refined and updated. These are categorized in my second brain with #garden / evergreen¹.

Andy Matuschak defines Evergreen notes as:

Evergreen notes are written and organized to evolve, contribute, and accumulate over time, across projects. This is an unusual way to think about writing notes: Most people take only transient notes.

That’s because these practices aren’t about writing notes; they’re about effectively developing insight: “Better note-taking” misses the point; what matters is “better thinking”. When done well, these notes can be quite valuable: Evergreen note-writing as fundamental unit of knowledge work.

Also, flagging as an evergreen note does not mean they are finished writing:

Taxonomy of note types
Evergreen ≠ finished.

All notes are alive, and continue to compound, these are Continuous Notes eventough categoriez as Evergreen note. Evergreen means the note is well grown and matured, but still growing.

Also, nothing as part of my Second Brain is ever finished, that’s the whole idea. Finished thoughts or none changing are published as blogs, as blog vs. notes explains further.

# Compounding Effect of Notes

It’s hard to write Notes that are worth developing and compounding over time. These principles by Andy Matuschak might help:

Evergreen notes should be atomic
Evergreen notes should be concept-oriented
Evergreen notes should be densely linked
Prefer associative ontologies to hierarchical taxonomies ^0dd872
- ? evergreen note is an Ontology? Therefore, Second Brains too?
Write notes for yourself by default, disregarding audience

# Zettelkasten

This concept is of course enormously indebted to the notion of a Zettelkasten. See Similarities and differences between evergreen note-writing and Zettelkasten.

Origin: Evergreen notes

Check out my Taxonomy of note types of all notes in my second brain/digital garden. ↩︎

'Having Taste' as an Artist—Age of AI

Mon, 05 Oct 2026 06:37:37 +0000

Having taste is key in the age of Generative AI.

# Having Taste as an Artist

It’s similar to a chef. A good chef will have the best taste and Creativity combined. As writers and artists, we need to use the same strategy and skills as a gourmet chef. Use taste as a differentiator from others.

Knowing what we want, but better, what we don’t. Same as always, Hell Yeah or No, thanks Derek Sivers (also what Wes McKinney concluded in Mythical Agent Month).

# Taste and Product Management: Experience

Also, my own learning from building a LOC (line of code) email client called Neomd with a solely agentic workflow, but still verifying every plan and following all the code changes manually. And most importantly, with a clear vision and plan for what the outcome should be, I added my taste, as Rick Rubin would say in a viral clip that he has “absolutely no ability, only adding his taste”:

The confidence that I have in my taste and my ability to express what I feel has proven helpful for artists.

A bit abstract, but this is what we have come to when we use agents: we need to add our taste. And how do we do that? With seniority and experience. The right foundations are still needed to guide and spot architectural errors in the model, or to detect if it drifts in the wrong direction.

Some say we need to be the product manager kind of type to guide the agents. I’d say yes, one part, but having experienced it, and also having a stable or rigid framework/structure of your project where the agents work in.

Even better if the structure is a modular setup, because then you can more easily add features without touching code at the heart of it and breaking it, I wrote more at Using an Existing Framework.

# The Future of Generative AI with Rick Rubin

Rick Rubin, as one of the most prolific music producers alive, says:

I know what I like and what I don’t like, and I’m decisive about what I like and what I don’t like.

The future of Generative AI might just to Having Taste (with AI), and hearing to your Gut Feeling, and make decision from the heart, like Writing from The Heart.

Rick Rubin doing his things: rick-rubin-doing-his-thing.webp :)

One example would be the email client, Neomd, I vibe-coded with my taste, dedicated features only I specifically have. Read on at Vibe Coding.

# Taste is Actually not Enough

Tom White at Rick Rubin Is the Future of Work says:

Unironically, Rick Rubin is the future of work.

You need creativity, insight, character - because Creativity is Forward Looking, and LLM are backward-looking. You can’t create a unique game from just copying old stuff.

# Pathless Path, not Planning Everything ahead

Follow the unconventional path, listeing to your Gut Feeling instead of data as Rick Rubin said when making decisions or going to school, we haven’t been trained to feel, or to express ourself freely. When I hear it, it sounds lot like the Default Path (gong to school, having a career, mary, house etc.) vs. the Pathless Path that Paul Millerd wrote about, to follow own path, not planning everything ahead.

# Others

Will AI Replace Human Thinking?
Rick Rubin
How to develop good taste: Good taste = a language. Good taste means speaking the right words so you can say what you’re wanting to say. Featuring Dave Grohl in the video.

Origin: Vibe Coding

Vibe Coding: Think Slow, Cultivate Taste, and Build Fast

Mon, 05 Oct 2026 06:31:09 +0000

Vibe coding is the next evolution of Prompt Engineering, when you instruct your AI Agents to run autonomously. The best example is Claude Code, but there are more.

# When Vibe Coding Works Best

Vibe coding is best when you have a framework around it. This is what we had with Data Warehouse Automation tools, and e.g., dbt.

We need to bring that back. Or like I use with GoHugo, it’s a predefined framework with rules and governance, so the LLM is very good at adding things, versus starting from Scratch.

Simon Whiteley with Advancing Analytics building something called LakeForge, which is essentially that too. The framework is the Declarative Data Stack, if it’s all code, the better. And LLMs can then extend and enhance.

It’s similar to Spec Driven Development (SDD), where we write our instructions in claude.md and Claude or any AI Agents implement them. Also, what Esco Obong said what they do at Airbnb, the hard part is coming up with the spec, talking to business, etc. The Coding part is small.

What you build is more important than how

“Don’t overrate how you do it; it matters more what you do, especially with LLM models. It’s just a tool”. Me on Tweet

If you build something quickly that took you hours, days, or weeks before, and can do it today in minutes, but it’s something nobody uses, you still wasted time and brain power. Jason Fried said the same in a different way: The how, what he calls ‘how many cooks’, is not the important part, but ’the menu’, the what they cook. His post at The speed at which a product is developed doesn’t inherently make the product better or worse. The number of commits, The …..

# Using an Existing Framework

I think a framework with clear rules and guardrails, like the declarative data stack, works best with agents, such as my website built with GoHugo. Building one from Scratch, AI is terrible at it (at least from a maintainability perspective), but from an existing Hugo setup, amazing.

I wrote a little more about « Data Modeling for the Agentic Era: Semantics, Speed, and Stewardship», which applies to any agent’s project, not just data modeling.

Note

Mario Zechner, creator of Pi (Agent Coding), calls it to build “modular” systems in Building pi in a World of Slop. He also say Slow down, learn to say no. Again, back to my favorite mantra, Hell Yeah or No, which also works amazing in real life.

An example of vibe coding and using a professional service for a website

As I always say Why have your Website, and vibe coding can help when you have an existing framework. Here’s a before after vibe coding article of a website: Your vibe coded website is going to get you fined

# Should We All Build Right Now?

I think. If you’re building right now. You should go hard all-in on what’s happening right now. Raphael Schaad. Tweet

Building, or just prompting? Not sure if that will lead us to a good place. To build for ourselves, sure, it’s fun. But for professional services, not so sure, yet.

But also, if you don’t use it, you get FOMO, and fear of being too slow. I think if management pushes AI down to developers as much, we will end up in a scenario like this:

I am done with this shit. It is over. The state of engineering right now is horrible. It has been half a month since I started a new role at a big company. Nobody knows anything here. The specs, code, tests, PRDs, tickets, resolution of those tickets, reports, etc., everything is…
— voxium (@v0xium) September 20, 2026

But pushing back is not as easy if management thinks AI solves everything.

# Craftmanship and 10x Developers

Is craftsmanship gone with AI? This excellent video goes into it: you can build something on demand, but you have no attachment to what you’ve built. So how are you gonna sell or rave about something you don’t even have any connection to? No hardship, no challenges, no learnings with AI.

Watch the video of I was a 10x engineer. Now I’m useless, or read Vibe coding and the death of craftsmanship.

This was also the message of PewDiePie’s video, Trains his own model, that he struggled so much that he learned so much. With vibe coding on auto-pilot, this is going away, fast.

# Vibe Coding Makes You Exhausted

AI making you Exhausted and potentially addicted. They makes us lazy. They make us do the Shallow Work, instad of the Deep Work.

AI turns bullet points into a nice text - oh look, AI puts this blob of text into bullet points - Meme

Great example: Generating an email that we sent do an actual human being | Source: AI Written, AI Read cartoon - Marketoonist | Tom Fishburne

Also, Id rather Read the Prompt.

# Soulless

AI agents don’t have souls, they have no characters. What’s the kick, the fun, the joy of it? This is also my take on Will AI Replace Human Thinking?, and how it is powerful and distractive at the same time, but lacking any depth.

# Shallow Work and Cost

Always be careful on Shallow Work vs real work you’re outsourcing to agents. Also highly related, Jevons Paradox, where the total cost of AI, because of everyone using it, is rising.

# Will Vibe Code Replace Actual Programmers?

Some even say vibe coding is killing open source - see How Vibe Coding Is Killing Open Source that says:

Generative AI is changing how software is produced and used. In vibe coding, an AI agent builds software by selecting and assembling open-source software (OSS), often without users directly reading documentation, reporting bugs, or otherwise engaging with maintainers. We study the equilibrium effects of vibe coding on the OSS ecosystem. (check its Whitepaper)

Read more at Will AI Replace Human Thinking.

A great quote:

Is writing code dead
Great Tweet where I got the inspiration from
Personal observation: Writing code by hand is obsolete. But real coding experience correlates really well with being able to direct agents. In other words, non-technicals cannot successfully evolve AI-assisted systems.

# Code Review in Times of AI

More than replacing, which is not the case any time soon, is the question of how to Code Review that can keep up with the amount of code that can be produced in minutes.

# Complexity Will Hit Us when Coding

See Simple Made Easy by Rich Hickey.

# How to Code with AI?

See How to Code with AI.

# Vibe Coding Tools

See Vibe Code Agents and AI IDE and AI Orchestrators.

# Speed

With Amdahl’s Law, when everything is only as fast as the total when you parallelize with its overhead, so if your tool takes many minutes to reply, you lose a lot more time than when the model will answer directly, even more if you automate and run them in parallel.

A write-up at I’m (mostly) picking models on speed now, not intelligence. And The other 90% that Never Tried Claude Code- Slow AI for Enterprise

# Thinking Slow: Cultivate Taste

See Slowing Down, and Having Taste (with AI).

Also in TikTokification, I say that:

Death of the Follower
Creativity and Art on the Internet with AI:
The goal of human Art and Craftsmanship is connection, not pure information. Humans take risks, doing something hard. The outcome will make you feel something.

Additional layer of Coding makes us slower, not faster

Apenwarr says on his blog about Every layer of review makes you 10x slower.

# Shipping and Figuring it out

Built a full CRUD app in 40 minutes last week. Spent three hours the next day figuring out what it actually did. PromptSlinger

Yes! What happens to me is that the thinking is before, I use the plan mode and go forth and back extensively. And then what it builds, I “usually” understand if it’s a project of mine (not from scratch, but with the framework) and Claude built what I wanted.

But I think that’s only possible where I know the things.

What do you think?

# What’s the Future?

I think The Future with Generative AI, is “Having Taste (with AI)”.

# The Most Important Skill

Besides Having Taste (with AI), some say that “in prompting is expertise in the domain you’re prompting for”. And I agree, if you don’t have domain knowlege, or don’t use the product yourself, the outcome will just be AI Slop.

We also discuss more at The Future of Creativity on the Internet in a World with AI.

# Vibe Coding Example

# ‘Make it Faster’ Prompt

I just (2026-09-30) did the infamous “make it faster” prompt on Neomd, and the outcomes where suprisingly good.

Claude just reduced the round trips for switch/screen/delete from 26 (all sequential, all blocking) to 0 blocking, with the rest running async behind the list, plus an in-memory folder cache so switching is instant. It’s magical, I will test more if all the features are still working, but so far so good.

But amazingly, Claude really got good with the auto-agents - I asked it to do a spec and implementation plan, make sure every feature still works and check them. It ran for 1h 57m 57s, and the outcome now is day/night—it’s blazingly fast now, instant! tbh, it was fast already before, crazy world. See also my tweet.

# New Features of Claude: Auto Review

A couple of new comments I didn’t see before from Claude Code upon finalization:

Final whole-branch review is running on the most capable model over all 15 commits. Nothing else is pending until it reports; then one fix wave if needed, then wrap-up.

So it uses different models on its own for review and even re-review:

Scoped re-review of the fix wave is running on opus. Its verdict decides whether residuals get parked with rulings or the branch is ready for your soak; nothing else is independent of it.

This is amazingly good for a simple app like mine. Before I used Roborev for this, now it comes all included.

# Upgrading My Project Requirements with Claude Code

I upgraded an old project of mine, practical.ssp.sh, with Claude Code, which is so cool, as you don’t need to do the hard work, and it just upgrades. On the other hand, I have no clue if all the changes it’s done are correct.

Sure, I can run the pipeline and assess more or less, if it still works. But all the details I’ve built with 100s of hours before, are they respected, or just ignored?

I pushed the PR to look at the changes later. I have almost no desire to look at it, although it’s so cool to see the latest version, quickly migrating from MinIO to SeaweedFS and adding back old steps.

But I didn’t learn anything. It’s fun, and not so fun at the same time. Very weird.

# Email Client for Markdown in the Terminal with Neovim

Neomd, a TUI for email reading in Markdown and writing in Neovim, fully vibe coded with heavily instructions from me.

# Testing and Quality of Vibe-Coding with Self-Experiment

Everyone wants to use AI to automate data work, but nobody wants to review, let alone maintain it. AI-generated code is very similar to written text, except that code is more functional. I did a little self-experiment to test the maintainability and threshold on when it starts to crack.

First, because I didn’t find a Markdown TUI email client, but 2nd also because I wanted to see how far I can go with a programming language I don’t know in details (Go) (see also Python Was Built for Humans, Not Agents), and vibe-coded my own email client, neomd.

I found that when the code is small and the prompt.md is clear, specifying your goal and referencing projects, all features are clear, with no workarounds or errors surfacing; everything is smooth and really great.

It starts with a larger code base, where you vibe code fixes that don’t account for the overall features or undo a fix you just did yesterday. And I must say, it’s really hard to let the AI model add a simple fix for something small, without breaking. Mine is currently at 11,029 lines across 25 files (excluding tests and docs).

Ah, and yeah, the tests are vibe-coded too, so do I trust them? Not really, I saw Claude just changing its tests all the time, instead of creating tests first and then fixing the code to match the test, it always does it the other way around.

Who doesn’t know the good old forth and back with AI? I asked to verify the safety of a change, as I saw too many changes and quickly checked it, and didn’t work - but AI thought otherwise 😉:

Me: Please verify safety
Claude: “Let me verify the safety […] Perfect! […] All tests pass, no Breaking Changes.”
Me: “But there’s an error, see X and Y”
Claude: “You’re absolutely right - this is broken!”

But then again, am I happy with my vibe-coded client? Yes, I use it daily and am trying to switch full-time. It’s a joy to use and to build if you know exactly what you want. But the hard part starts now, keeping it working for a long time. I will report back on how it goes. And it’s a very good experiment to get a feel for how vibe coding is as of now. What’s yours?

Ah, and if you are interested in testing it or seeing the tests or code it created, you can check it out on GitHub.

# Learning? Good Architecture Matters

Good architecture matters, especially decoupled architecture.

I notice the specs are getting too big for the model to read each time. I have all the specs written down as docs (markdown) with all the features, etc., and a CHANGELOG with all the changes. Still, there’s a point where the details and overall architecture are just too big for the model (and the probability predictor) to handle.

Docs (specs if you want) are 2,724 lines across 16 files, and tests are 5,667 lines across 15 files, and I feel it cannot handle it anymore. But that’s probably also depends on the structure.

On my website, also GoHugo, I have a much larger codebase, but the framework lets the model work in dedicated areas without breaking others. Which, in this vibe-coded project, is not the case. I guess poor architecture as vibe coded from the start!

This is exactly the point I made Using a Framework earlier, and it’s verified once again by this experiment. And Test Driven Development high-quality, manually written test will get more important than ever.

# Other Example: Codebase Disaster after 6 Months

Shipping is fast, maintaining is a nightmare. Changing something here, something else breaks. I’m sure these kind of stories will be all over:

Yo Claude, rewrite my vibe-coded app: > make it clean. > think robust architecture. and most importantly, make no mistakes or else… 🤖🔫 | Michael Kisilenko | 85 comments

# Token Usage

It’s changing, more companies monitor and restrict due to high cost. Great break down by Gergely Orosz at The Pulse: token spend breaks budgets.

# Vibe Coding ≠ Vibe Coding

Wes McKinney made it clear in an interview with me, that what he does, it not vibe coding, but does Agentic Engineering. All the planning, architecting with superpowers and his tools, it’s not the same as vibe coding.

The future will not be buy vs build, but buy vs. build vs. vibe code, ThePrimeagen says at tokenmaxxing talk. AI Token Efficiency will be a thing too.

# The Hype Cycles of AI

So many hype terms

The Hype Cycles of AI
The cycle of hype terms is going strong:
chatting, plan mode, tokenmaxxing, loop engineering, harnesses, graphs, orbs, and terms will continue

The Hype Cycles of AI
Others that nobody talks about anymore are:
Moltbot/OpenClaw

What’s next?

# Majurity of Vibe (Different Levels of Programming)

Different levels of programming code, as Karim Jedda says that Vibe coding is a bad term, and it’s better to look from the accountability angle:

A0: human authored, with AI at most as a search engine or sounding board

A1: AI assisted, autocomplete, snippets, boilerplate etc but the human is the author

A2: Agent built and human-reviewed (ie: the dev can explain every line and is accountable as if they’d written it)

A3: Agent built and unreviewed (which is what “vibe coding” initially described), good for prototypes and throwaway tools. This contains agent built & agent reviewed.

# Further Reads

Some even say vibe coding is killing open source How Vibe Coding Is Killing Open Source. See also related Whitepaper.
Self-Improving Software | Jeff Lunt
AI Tooling and Open Source handling PRs in Apache Arrow
The latest on Vibe Coding with Claude by Andrej Karpathy
Token Anxiety
Use Given/When/Then Specs to Make AI Generate Production-Ready Pipelines, Not Spaghetti Code – Start Data Engineering by Joseph Machado. Behavior-Driven Development is used with Gherkin.
Agentic Coding is a Trap | Lars Faye
How data practitioners use Vibe coding in the Joe Reis survey with 1101 participants: 2026 State of Data Engineering Survey | Practical Data Community

Origin: LakeForge, Tweet

Death of the Follower: TikTokification

Mon, 05 Oct 2026 06:30:16 +0000

The death of the follower is more relevant than ever, even though the talk was given two years ago. Today, even LinkedIn and many other platforms are TikTok-ified.

Basically, all Social Media Algorithms have changed, so you can’t reach your followers anymore. It started with TikTok, but with the latest LinkedIn updates, it’s the same there, and on YouTube as well.

Jack Conte explains it so well, and created Patreon therefore. His Death of the Follower & the Future of Creativity on the Web talk at SXSW 2024 Keynote - the AI Highlights from YouTube:

Creator led the community to the platform first:

It’s hard to reach your followers. It’s defined by the Algorithms

In 2010, when Google ranked our website (and still does today), and in 2020, with the death of the follower, we no longer reach our own followers. It’s what the TikTok algorithm introduces and is now spreading everywhere. It’s the TikTokification.

And today (2026), the AI chatbots are another beast, stealing a writer/blogger’s intellectual property and sharing it without anyone ever visiting our website.

Bluesky is different

If we’d been using a network like Bluesky, where you actually see what you follow, and no algorithm decides, it’s actually still pretty OK. You can actually follow the users. Plus, it’s built on an open protocol called AT Protocol.

# The Birth of the Followers

The birth of the followers was YouTube Jack said. People were seeing what you upload, every time.

And with that, according to Kevin Kelly you needed (1000) True fans to make a living. Today, if we can’t reach our 1000 followers, so it doesn’t matter anymore how many we have and is more of a Vanity Metric.

Source

# Creator Economy

Source: Death of the Follower & the Future of Creativity on the Web with Jack Conte | SXSW 2024 Keynote - YouTube

Bluesky
Gumroad
Discord

AI Summary of the full video

The internet has evolved from Web 1.0 (read-only) to Web 2.0 (interactive) and is now potentially entering a new phase where the concept of the follower is weakening.

The speaker started making music and videos in 2007, leveraging Web 2.0 platforms like YouTube to reach an audience.

The “subscribe” button was foundational in Web 2.0, enabling creators to build a following around their work.

Kevin Kelly’s concept of “1000 True Fans” emphasizes the importance of a core group of dedicated fans who support creators through purchases and engagement.

Social media algorithms and platforms like TikTok prioritize engagement and attention, potentially breaking the direct connection between creators and their followers.

The speaker expresses concern about the “death of the follower” and the challenges creators face in reaching their audience and building community.

Patreon is evolving from a membership platform to a comprehensive platform focusing on media, community, and business, aiming to strengthen direct-to-fan connections.

Patreon’s new features include community spaces, commerce tools, free membership options, and live experiences through the acquisition of Moment, all designed to foster deeper fan engagement.

Creators should invest in true fans, make beautiful things that resonate with them, and stay true to their artistic vision and goals.

The speaker emphasizes the importance of knowing what you want as an artist and not letting external metrics or algorithms dictate your creative output.

# The Future of Creativity on the Internet in a World with AI

The future of creativity on the internet in a world with AI by Jack Conte. And the future of AI is the Creativity of real humans. Making real Art. As I wrote in The Act and the Outcome of Creation. And at The Future with Generative AI with having Taste (with AI).

Creativity and Art on the Internet with AI:
The goal of human Art and Craftsmanship is connection, not pure information. Humans take risks, doing something hard. The outcome will make you feel something. ^ef7f59

# Further Reads

Well Being in Times of Algorithms (obs)
Open Social with AT Protocol and Bluesky as an example app, see Tangled as another one.
Social Media Addiction
Surveillance

Origin: Jack Conte on Death of the Follower & the Future of Creativity on the Web with Jack Conte | SXSW 2024 Keynote - YouTube
References: Social Media Addiction, Stolen Focus- Why You Can’t Pay Attention (Johann Hari), Birth of the Followers

The Pathless Path

Mon, 05 Oct 2026 06:25:14 +0000

The Pathless Path, introduced as a book by Paul Millerd, is the unconventional path you follow your Intuition and do things that are not common or planned. Unlike the Default Path which is what usually we do with going to school, having a stable job, mary someone, have a house, and kids. Not having a plan shouldn’t be a goal, but not everything should be planned out ahead.

The Pathless path is also experimenting on your own to find a life you want to live vs. “getting ahead” on a script others follow with embracing uncertainty & discomfort. I strongly believe that the urge to follow the default path and continually climb the ladder is also not the right thing to do. Better follow your Instinct, the pathless route sometimes. ^cf4966

The book and the term Pathless Path inspired me to write my own story about my life, I called it Finding My Pathless Path.

My Pathless Path

I went out of comfort early on, here the short version of my path so far:

applied for a job in another canton

went to Australia for 15 weeks by myself

lived by myself as opposed to joining a shared flat

after 4 years, yet again out of comfort being a consultant 4 years

studying on the side while working as a consultant 90%

living in a foreign country

applying for a job in English

changing job to be a writer and 100% remote

A different Framing

The worst plan for the next two years, the best for the next 10. David Perell on Becoming A Citizen of The Internet about following his path or Intuition, and building a cohort around creating ideas.

Anything You Want (Dekerk Sivers) says:

Most people don’t know why they’re doing what they’re doing. They imitate others, go with the flow, and follow paths without making their own.

# Further Reads

Paul Millerd on How to Quit Your Job (and Find Work You Actually Love) with Ali Abdaal

Origin: Pathless Path is from the book The Pathless Path (Paul Millerd).

anxiety

Mon, 05 Oct 2026 06:15:53 +0000

Don’t forget, we are all human.

It’s normal to have anxiety, not to know what comes next, to have negative emotions, to feel overwhelmed. When that happens, try to slow down. Say no to all things. Go out in Nature, be present with your family, and recharge!

Also, follow your instincts:

Pathless Path
The Pathless path is also experimenting on your own to find a life you want to live vs. “getting ahead” on a script others follow with embracing uncertainty & discomfort. I strongly believe that the urge to follow the default path and continually climb the ladder is also not the right thing to do. Better follow your Instinct, the pathless route sometimes.

Origin:
References: Reflections

How to Keep Enjoying the Writing (And Keep Going)

Sun, 04 Oct 2026 08:57:27 +0000

To keep going with Writing (I just learned that I am online for 22+ years plus, and sharing my first blog since 2015), if it weren’t for the learning, I guess I couldn’t do it anymore. But because I enjoy it and can learn something, as you say too, that’s why it can last, even if the outcome might be only for yourself.

To me, Paul Millerd says it best in his Pathless Path book, but also says it in this video in regard to writing: Six Unconventional Rules Of Writing. Paul says that he can’t play the game of the Algorithms, that would kill his creativity, the exact same philosophy I have too.

Learning every day is one of my first principles.

# Futher Reads

Origin: Paul Millerd with Six Unconventional Rules Of Writing - YouTube, and comment on Substack

Keep AI Out of Your (Obsidian) Vault

Sun, 04 Oct 2026 08:43:55 +0000

Everyone is using Obsidian for AI, or wants to use it to become more productive. But I think it’s a dead end.

The reasons are clear: Obsidian notes are open on your local disk in an open format, Markdown, and accessible to everyone, including your AI agents. This makes it as great and easy an integration as possible. But should you actually use AI with your notes? And if so, what are the use cases or ways you should interact with it?

First of all, if you use Vibe Code Agents with Obsidian, use the Obsidian CLI. It will be much faster to return your files, search, or act, grepping through the files.

Second, I’m a strong proponent of avoiding adding lots of AI-generated summaries[^1] or other on-the-fly-generated text to my vault. The reason is simple: over time, I don’t know anymore whether the content was written by me or by an AI, and my own, much more valuable thoughts get diminished by “AI Slop”.

Also, when searching for something, you need to fight through the noise of generated stuff. If you only have your own writing, it’s all valuable, or at least there’s a reason why you noted it down. There was a conviction, idea, or something that moved you. ^90383c
Yes, it’s good in certain areas, and mostly on-the-spot summaries seem great, but every time I come back to them, I find them very average. It doesn’t help me, as it didn’t highlight the things I would, and I need to reread anyway.
What I do sometimes with Obsidian Webclipper, if I create a new note, is summarize it in one sentence or a few. I clearly mark it as AI-generated and even put it in a quote, so it’s clear to me, and to anyone in 5 years, that it wasn’t mine. If I have added my own thoughts and writing, I will just remove the generated paragraphs (not my words).

But, use it for advanced research, like finding related notes. I think this might be the best use case. But don’t use it for tagging or organization, because eventually all your relations and connections won’t count for anything, since they aren’t made by you. The power lies in the deliberately created graph of notes, your very own Second Brain.

My Obsidian Vault with Smart Lookup and related notes with Smart Connections and local graph | Also check out Vim Motions for Writers to see my writing workflow and how I write

I personally don’t use it like that yet, but I might one day with a powerful local model, since I have lots of sensitive notes I don’t want to upload anywhere. Plus, it takes away the thinking part of My Obsidian Note-Taking Workflow. I don’t want to replace my human thinking yet. But I use a mathematical or local model with Obsidian Smart Connections, or the Graph Analysis plugin.

I’m pretty sure that if you go all in with using AI to generate your notes, you will end up with less clarity and fewer insights. You might start fresh very soon, as you won’t get value from it, and stop maintaining or adding ideas.

[^1] To be clear, I mean long summaries, no harm in a 1-2 sentence intro in a note.

Long AI Summarization is usually just noise

Kepano says, too: “A summary of a PDF is noise. An insight I had from reading the PDF is signal”. Tweet. Read more Other Opinions by Jason Fried, Paul Graham, Ted Gioia, Mitchell Hashimoto, and many more.

# Search is an Organizational Question

I think if you have an urge to do something, do it in a separate vault, or just use a database, e.g. DuckDB, and do all the fun stuff with AI as I did for search and clustering. But don’t mix your precious notes and Zettelkasten for it. Not yet, at least, but probably never.

I have 25'979 files totaling 3.5 GB, all in a single vault, and I find everything in split seconds (using the Omnisearch plugin). Search is pretty much an organizational question; e.g., I use Zettelkasten, and some refer to Memex as its digital version.

But if you need more advanced features, you can just add the Obsidian Smart Connections plugin and get vector and similarity search, etc., out of the box.

# Your Notes Will Be Your Prompts Tomorrow

Remember, your notes will be prompts or libraries for AI tomorrow, but not if you generate them.

# Cultivating Knowledge

Also on the notion of cultivating knowledge for AI systems, I’d say I’m more interested in cultivating for myself, long-term 🙂.

I imagine if humans do not have their own knowledge bases, everyone will have the same (average) notes based on AI models, not 100% of course, but there is no conviction, no decisions made, just all equally similar.

This is another reason we also need human-curated knowledge on a large scale. AI models will retrain on this data, better than on synthetic, fake data, IMO, though we need to find an economic model that also works for the creator of these insights and knowledge.

# Whose Knowledge System is It?

Using Claude Code (or another agent) to generate your system and make connections, like , is not your system. If you want a tool for retrieval, yes, sure, but I have a second brain and a knowledge system where my thinking happens. I make the connections because I had an idea, or something told me to do so.

Great video on this with more of the same thinking I have at Obsidian vs Claude — Why I’m Not Worried About AI Killing Note-Taking Apps. He says elegantly that a second brain is a system; the connections and writing parts are not inefficient; it’s the process of thinking and learning.

Claude is just a tool, like Obsidian, but as File Over App, by the CEO of Obsidian, goes, it’s not about the app anyway, it’s about the files, your thoughts, your insights. So if it’s someone else’s, heck, even generated, it’s not why I use the second brain. Can be useful, too, sure, but not as much as my new ideas, the thinking I get from my knowledge system.
Touch some grass, they say.

# Examples

Some examples that identify this phenomenon quite well, or help us understand and think through it better.

# It’s Hard to Finish an Idea that is not Yours

Its Hard to Finish an Idea that is not Yours (and suggested by AI)
While I use Claude Code a lot for my Omarchy plugins and Linux optimization/failure detection, I don’t use it for writing and note-taking, especially for coming up with unique ideas and driving them home by writing them through to the end.

Its Hard to Finish an Idea that is not Yours (and suggested by AI)
It’s so hard to finish an idea that is not yours and is just suggested by AI. Instead, I like to Write Manually, Daydream, and follow my own train of thought, even if not all-encompassing, but at least it’s my thought, the idea that has an interest and connection to me.

# Andrej Karpathy’s LLM Knowledge

Andrej Karpathy LLM Knowledge Auto Research, which is another good example of how not to do it (IMO), especially if you like to learn. Sure, it’s fun and helpful, but it will lose its appeal and value over time. It’s like this Reddit quote says it well:

I definitely agree that the ‘Karpathy LLM Wiki’ mediated knowledge base everyone is excited about is not a useful way to actually learn or manage your learned and curated data for the reasons written. If instead you do just want an automated research summary of some knowledge - sure why not, but it’s not particularly interesting and I really don’t get the excitement about it. Reddit

Some great discussion in this thread on this topic.

# Does not Scale, or Will Die Very soon

Automating the links and connections like this won’t sustain for YOUR second brain, for your ideas and notes - you are skipping the thinking, and IMO, the insights. Most of my ideas come from linking and seeing the links I made myself, sometimes years back.

A good example of what not to do for your own vault. Tweet

For sure interesting to see what comes out, but I wouldn’t recommend it for your “second brain” where you develop your thinking and YOUR ideas.

If you want the above brain, which is not yours, you should use Wikipedia. Wikipedia graphs and wikilinks give you great insights, as the automated AI ones (probably more), but made by human collaborators. But if you want to create your own knowledge base, I’d do it manually. The payoff comes over the year, it compounds.

# PARA Method: Dedicated Folder

The PARA method by Tiago Forte is super convenient - and you could create an AI folder in resources and put all generated notes there and hide that from search or through other mechanisms. This way, you could still have AI-generated insights or notes that you can link, if helpful, to your original notes.

# Voice to Text

AI with Voice to Text can be another approach to use with Obsidian, which makes sense when you are in a car or so.

But because of my talking much more compared to when I write, and to AI or translator service not understanding all my words, it’s much more work for me than just typing it out. Plus, I usually skip the thinking, though sometimes talking it through can add to the process.

# Slowing Down: Have More Impact Long-term?

See Slowing Down.

# Creativity by Its Very Nature is Forward-looking

Creativity is Forward Looking

# Further Reads

Origin: Li by mehdio, Obsidian
References: Writing is Thinking

The Problem is not the AI Code, but Nobody Knows Anything Anymore

Sat, 03 Oct 2026 19:08:54 +0000

If we think writing code is dead, and AI is generating all codebases, I still think the bigger problem is people or full teams not knowing anything anymore about the system architecture or the intent behind why certain choices have been made.

A comment on a discussion I had:

I think AI writes probably average code (depending on the task and size). So if your code base was below average AI can easily improve it up to average. At least that’s what I’ve observed here.

To me, the problem is not the AI code, but that nobody knows anything, and everyone just asks Claude. You end up with no plan whatsoever.

# The Current State in Fast-Moving Startups

This tweet summarizes the current state at fast-moving startups well, or larger companies where middle management is pushing AI hard:

I am done with this shit. It is over. The state of engineering right now is horrible. It has been half a month since I started a new role at a big company. Nobody knows anything here. The specs, code, tests, PRDs, tickets, resolution of those tickets, reports, etc., everything is made by Claude Code.

Nobody on my team likes this. They are being forced to ship as much as they can. I have heard multiple times from higher management that pushing code is not a bottleneck, so why are we slow? People are working 12 to 13 hours a day just to press enter. Nobody is reading anything. Humans in corporate are doing nothing on their own.

Everyone, literally everyone, from an L1 to an L7 engineer here is doing the same thing. Talk to Claude. There is no sense of victory. Nobody is resolving bugs. In reality, nobody is thinking anymore. Everything is done by LLMs.

It is so soul-sucking. I would not mind it, to be honest, if we were at least given the time to check out the code and see what is going where. But no, the goal is to just ship. No matter what happens. Voxium

Matthew Mullins says it well too:

We currently have a problem with COBOL programmers leaving the workforce, but agent coding is going to bring that problem to every development language.

# A Product Manager Could Now Build Anything He Wants

One could say a good product manager could now build anything they want and find a market, make it look good, etc., as the good point by Sean Behan:

I’ve always admired product people who can’t code but can manage a team to get the software they want. Knowing what you want has always been the hardest part.

But then again, if you can’t code, you will essentially build a very bad foundation for a product that’s very hard to maintain (although AI is getting better at that too, especially when you iterate often, but still, if you choose the wrong language or the wrong mental model, you have the wrong start from the get-go).

It still helps to know the fundamentals, either way: for programming and designing a product, and for a good PM who knows what is needed but also understands system and architecture design.

# What about other Fields, like Data Engineering?

Hoyt Emerson mentions that data engineering is different:

I think Data people are different. We’ve had to know everything about the product/business from day 1. AI just removes friction for us now. Tweet

I think data people who grew up pre-AI had to know everything (or a lot, or involve domain experts) to figure it out, indeed. But AI makes this obsolete, or seemingly obsolete.

That’s why people starting today, or me as well, if I start today prompting away in a new field, all of a sudden, that knowledge is missing.

# The Final Boss is Still Maintenance

Thinking in systems, or architectures, or having intent and design- all of them help to be a better software engineer. Nowadays, Writing code by hand might be dead, but it certainly helps, and Having Taste (with AI) is more important than ever.

But the final boss is, and always will be, maintainability. The easier it is to generate a quick pipeline, app, or BI dashboard, the more you have to maintain. And if nobody knows a thing, that can get really hard.

# AI Can’t Drive Itself

Yes, the AI can’t prompt itself, right? Why do we even need humans? To me, it’s a clear sign that humans are still needed to direct and orchestrate it. That’s also why intent, taste, design, and architecture are all killer features in today’s world.

But once these are absent, or even worse, fundamental, get lost, it’s really dangerous. I read today that this is a self-inflicted problem, and if we still hired juniors, then the problem wouldn’t be happening. But yeah, it’s not as easy.

On Intent and Conviction

Harry Dry said it early on that:
Will AI replace Humans
Big ideas are less about creativity and more about conviction. [..] So, what happened? ‘Sauce and other shit’ got incredibly cheap! [..] There is no AI prompt for conviction. Harry Dry

I wrote more on Is AI solving this? that writing is hard, and writing from the heart is something only humans can do.

# Further Reads

Kris Jenkins talking about the worry of middle management, not the vibe coding at Danger of AI or LLMs
If Consumed by a Human, Should Be Written by Human
What I Learned Writing with AI
Vibe Coding

Origin: ThePrimeagen video and Limitations of LLMs and AI
References: What I Learned Writing with AI

GoHugo

Sat, 03 Oct 2026 16:10:22 +0000

The world’s fastest framework for building websites. Hugo is one of the most popular Open Source Static Site Generators (SSG). With its incredible speed and flexibility, Hugo makes building websites fun again.

It’s built with GoLang.

# Why I Use it

I wrote my journey from WordPress to Hugo in Saying Goodbye to WordPress.

Also check Public Second Brain with Quartz, where I use Hugo to publish my Obsidian notes as a public second brain.

# Migrated from OneNote

Migrating from WordPress to Hugo

# Docs

Similar to Docusaurus, which focuses only on Documentation, GoHugo offers themes specific to docs. Docsy, which might be the best one made by Google.

# Alternatives

Static Site Generators (SSG)

Origin: My Websites

Migrating from WordPress to Hugo

Sat, 03 Oct 2026 16:09:01 +0000

# How the migration worked

I migrated as quick as I did because of how convenient it was to export the entire content I had written over time. I used wordpress-to-hugo-exporter, which is one of the provided plugins on GoHugo. After exporting, I had all my post as single markdown files and all images linked to them.

You find more on Saying Goodbye to WordPress, a Homecoming.

# Alternatives

After searching for Python-based frameworks, I quickly found topics around Static Site Generators (SSG) and with it GoHugo. Without more extensive research, I just started using it and get a feeling for it. Below are the first steps which keep you going with GoHugo.

# Get started with GoHugo

brew install hugo
#load example blog
hugo new site quickstart
cd quickstart
git init
#adding a template
git submodule add https://github.com/theNewDynamic/gohugo-theme-ananke.git themes/ananke
echo theme = \"ananke\" >> config.toml
#adding a new post
hugo new posts/my-first-post.md

To start seeing your build website live on your local machine, you start the webserver, which automatically updates every change. And within seconds, that’s also where GoHugo seems to have an advantage over other SSGs, which are fast on production but not as quickly to render locally.

hugo server -D
                   | EN
+------------------+----+
  Pages            | 10
  Paginator pages  |  0
  Non-page files   |  0
  Static files     |  3
  Processed images |  0
  Aliases          |  1
  Sitemaps         |  1
  Cleaned          |  0

Total in 11 ms
Watching for changes in /Users/bep/quickstart/{content,data,layouts,static,themes}
Watching for config changes in /Users/bep/quickstart/config.toml
Environment: "development"
Serving pages from memory
Running in Fast Render Mode. For complete rebuilds on change: Hugo server --disableFastRender
Web Server is available at http://localhost:1313/ (bind-address 127.0.0.1)
Press Ctrl+C to stop

This produces this static website which can live opened and edited on localhost:1313.

Origin: Saying Goodbye to WordPress, a Homecoming | ssp.sh
References: GoHugo, OneNote

Data Engineering Acquisitions (2022-2026)

Fri, 02 Oct 2026 21:46:23 +0000

Consolidation in the Data Engineering market is happening quickly. Tools from the Modern Data Stack get unified into bigger Data Platforms. This note highlights the latest acquisitions across data engineering. It serves as an overview of the latest consolidations.

Find attached the acquisition overview from 2022 to today.

gantt
    title Data Engineering Acquisitions Timeline 2022-2026
    dateFormat YYYY-MM
    axisFormat %b %Y
    
    section Fivetran + dbt
    dbt → MetricFlow                     :milestone, 2023-02, 0d
    dbt → SDF                            :milestone, 2025-01, 0d
    Fivetran → Census                    :milestone, 2025-05, 0d
    Fivetran → SQLMesh                   :milestone, 2025-09, 0d
    Fivetran → dbt                       :milestone, 2025-10, 0d
    Fivetran → Great Expectations (Steward)         :milestone, 2026-05, 0d
    
    section Databricks 
    Okera                   :milestone, 2023-05, 0d
    MosaicML                :milestone, 2023-06, 0d
    Arcion                  :milestone, 2023-10, 0d
    Tabular                 :milestone, 2024-06, 0d
    Neon                    :milestone, 2025-05, 0d
    
    section Snowflake 
    Streamlit                :milestone, 2022-03, 0d
    Applica                  :milestone, 2022-08, 0d
    SnowConvert              :milestone, 2023-01, 0d
    LeapYear                 :milestone, 2023-02, 0d
    Neeva                    :milestone, 2023-05, 0d
    TruEra                   :milestone, 2024-05, 0d
    Datavolo                 :milestone, 2024-11, 0d
    Crunchy Data             :milestone, 2025-06, 0d
    Select Star              :milestone, 2025-11, 0d
    
    section IBM + Confluent 
    IBM → Confluent         :milestone, 2026-03, 0d
    Databand.ai       :milestone, 2022-06, 0d
    Immerok                  :milestone, 2023-01, 0d
    WarpStream               :milestone, 2024-09, 0d
	
	
    section Qlik 
    Talend                        :milestone, 2023-01, 0d
    Kyndi                         :milestone, 2024-01, 0d
    Upsolver                      :milestone, 2025-01, 0d
	
    section Analytics & BI
    Alteryx → Trifacta                   :milestone, 2022-01, 0d
    Hex → Hashboard                      :milestone, 2025-04, 0d
    Coalesce.io → CastorDocs             :milestone, 2025-03, 0d
    MotherDuck → Tower.dev             :milestone, 2026-08, 0d
    AWS → DuckLabs            :milestone, 2026-08, 0d
    
    section Data Quality & Observability
    Bigeye → Data Advantage Group        :milestone, 2023-06, 0d
    Datadog → Metaplane                  :milestone, 2025-04, 0d
    Soda → nannyML                       :milestone, 2025-06, 0d
    
    
    section Streaming & Real-time
    Cloudflare → Arroyo                  :milestone, 2025-04, 0d
    Redis → Decodable                    :milestone, 2025-09, 0d
    
    section Database & Infrastructure
    NetApp → Instaclustr                 :milestone, 2022-04, 0d
    Vector Capital → SingleStore         :milestone, 2025-09, 0d
    Supabase → Hydra             :milestone, 2026-02, 0d
    Supabase → Turso            :milestone, 2026-10, 0d
	
    section SAP
    Dremio         :milestone, 2026-05, 0d

^4d105f

# Timeline

The timelines show each year’s acquisitions, starting in 2022. After that, we discuss related topics such as Bundling vs. Unbundling and the general state of the data engineering ecosystem.

# 2026

2026-10-02: Supabase -> Turso
- Announcement: Tweet - i’m been a huge fan of the team and what they’ve built. we have big plans together. more to share in our keynote in a few hours
  - Turso is joining Supabase to give every agent its own database
  - Press: Supabase Announces $150M in New Funding and Turso Acquisition
2026-08-26: AWS -> DuckLabs
- Announcement: AWS to acquire DuckLabs, the Amsterdam-based company behind DuckDB. We are not acquiring the DuckDB open source project, which will remain free and open source under the independent DuckDB Foundation (the non-profit that oversees DuckDB) and available under the MIT license as it does today. ^319c65
  - DuckLabs: DuckLabs – DuckLabs to Join AWS, Projects to Remain Open Source
    - What will change: “Joining AWS will give us greater capacity to invest in both the technology and the community around it. Going forward, the DuckDB Foundation will include a technical advisory board, so that leading community members can provide their input on the project’s technical direction. We also plan to open the extension stack so that extensions signed by other developers and organizations can run in DuckDB.”
  - AWS: AWS to acquire DuckLabs, the company behind DuckDB
  - Mike Driscoll on Why AWS Bought DuckLabs.
2026-08-25 MotherDuck -> Tower.dev
- Announcement: MotherDuck acquires Tower.dev
2026-07-13: Prefect -> Dagster
- Announcement: Prefect acquires Dagster Labs, Nick Schrock ^04bb81
  - See more differences between Dagster and Prefect (and Airflow) with and FAQ about the merger
2026-05-08: SAP -> Dremio ^44670f
- Announcement: SAP to Acquire Dremio (The transaction is still pending regulatory approval.)
  - With Dremio, SAP will deliver a universal, Open Catalogs built on Apache Polaris and the open Apache Iceberg REST Catalog API. It serves as both the discovery and semantic layer of SAP Business Data Cloud, giving every connected engine – SAP or non-SAP – a single point of access to unified business context: meaning, relationships, access rights and data lineage. This catalog will form the foundation of the SAP Knowledge Graph, embedding business relationships, organizational hierarchies, regulatory classifications and cross-system lineage as native properties.
  - 2026-07-06: Complete: SAP Completes Dremio Acquisition
2026-05-07 Undisclosed Company -> Great Expectations (Fivetran didn’t acquire, but becomes steward of open source community and GX Core project)
- Annoucement: Great Expecations, Fivetran
  - GX Core will continue as an open source, community-driven project. Fivetran intends to support ongoing project maintenance, ecosystem integrations, and community engagement.
2026-03-17: IBM -> Confluent^e4f448
- Announcement: 2026-03-17: Finish Acquisition, 2025-12-08: IBM start to aquire
2026-03-19: Astral (The company behind uv, ruff, ty) -> OpenAI
- Announcement: Astral: Astral to join OpenAI, OpenAI: OpenAI to acquire Astral | OpenAI
2026-02-10: Supabase -> Hydra
- Announcement: Hydra joins Supabase

# 2025

2025-12-16: Matatika -> Meltano (Arch) ^61399b
- Announcement: LinkedIn Post
2025-11-24: Snowflake → Select Star
- Announcement: Select Star Joins Snowflake: A New Chapter for AI + Data Discovery and Snowflake to Acquire the Select Star Technology to Expand Horizon Catalog’s View of Enterprise Data for Next-Gen AI
2025-11-18: DVC → LakeFS ^c38a54
- Announcement: A Celebration of Shared Vision: lakeFS + DVC
2025-11-04: ClickHouse → LibreChat
- Announcement: ClickHouse welcomes LibreChat
2025-10-13: Fivetran → dbt ^746334
- Announcement: dbt Labs + Fivetran: Open data Infrastructure for analytics and AI | dbt Labs
  - latest updates at dbt
2025-09-10: Vector Capital Management → SingleStore ^b9f630
- Announcement: SingleStore Announces Growth Buyout Led by Vector Capital
2025-09-04: Redis → Decodable ^413719
- Announcement: Redis to acquire Decodable
2025-09-03: Fivetran → SQLMesh ^08063d
- Announcement: Acquisition (SQLMesh)
2025-06-13: Soda → nannyML
- Announcement: Announcement
2025-06-02: Snowflake → Crunchy Data ^193bc3
- Announcement: Announcement
- Deal Value: Estimated the deal to be worth around $250 million. Delivering the Most Enterprise-Ready Postgres, Built for Snowflake
2025-05-14: Databricks → Neon ^3cf8ff
- Announcement: Announcement
- Deal Value: Bought Neon for a reported $1 billion in equity ( source)
2025-05-01: Fivetran → Census
- Announcement: Fivetran Signs Agreement to Acquire Census
- Blog: Why Fivetran and Census are joining forces
2025-04-30: Hex → Hashboard
- Announcement: Hex Acquires Hashboard
- Blog: Hex has acquired Hashboard
2025-04-23: Datadog → Metaplane
- Announcement: Datadog acquires Metaplane
- Blog: Metaplane by Datadog
2025-04-10: Arroyo → Cloudflare
- Announcement: Arroyo is joining Cloudflare | Arroyo blog
- Comment: To build R2 SQL
2025-03-19: Coalesce.io → CastorDocs
- Announcement: Coalesce Expands Data Platform With CastorDoc Acquisition
- Blog: Building the Future of Data Together: Why Coalesce Acquired CastorDoc
2025-01-14: Qlik → Upsolver
- Announcement: Qlik Acquires Upsolver
2025-01-14: dbt Labs → SDF
- Announcement: dbt Labs Acquires SDF Labs
- Blog: dbt Labs acquires SDF Labs to advance analytics engineering

# 2024

2024-11-20: Snowflake → Datavolo
- Announcement: Snowflake Agrees to Acquire Open Data Integration Platform Datavolo
- Blog: Snowflake snaps up data management company Datavolo
- Deal Value: Approximately $110M (primarily stock, remainder cash)
- Comment: Powered by Apache NiFi, provides single platform for automating and managing both structured and unstructured data flows
2024-09-09: Confluent → WarpStream
- Announcement: Confluent Acquires WarpStream to Advance Next-Gen BYOC Data Streaming
- Blog: Confluent Acquires WarpStream
- Deal Value: $220M
2024-06-04: Databricks → Tabular (Iceberg) ^2f6984
- Announcement: Databricks Agrees to Acquire Tabular
- Blog: Databricks + Tabular ^6a38ee
2024-05-22: Snowflake → TruEra
- Announcement: Snowflake Acquires TruEra to Bring LLM & ML Observability to Data Cloud
- Blog: Snowflake acquires TruEra to deliver LLM observability inside Data Cloud
- Comment: Provides tools to test, debug, and monitor ML models and LLM apps in production
2024-03-19: Databricks → Lilac AI
- Announcement: Lilac Joins Databricks to Simplify Unstructured Data Evaluation for Generative AI
- Blog: Databricks acquires Lilac to supercharge data quality efforts for gen AI apps
2024-01-30: Databricks → Einblick
- Announcement: Welcome to the Data Intelligence Platform, Databricks + Einblick
- Blog: Databricks acquires team behind natural language data notebook Einblick
2024-01-18: Qlik → Kyndi
- Announcement: Qlik Acquires Kyndi to Advance AI-Driven Business Outcomes

# 2023

2023-10-23: Databricks → Arcion
- Announcement: After 43B valuation, Databricks acquires data replication startup Arcion for 100M
- Deal Value: $100M
- Comment: Data replication
2023-06: Databricks → MosaicML
- Announcement: Databricks Completes Acquisition of MosaicML
- Deal Value: $1.3B
- Comment: One of the biggest AI deals of 2023
2023-06-22: Bigeye → Data Advantage Group
- Announcement: Data observability platform Bigeye announces acquisition of Data Advantage Group
2023-05-24: Snowflake → Neeva
- Announcement: Snowflake Acquires Neeva to Accelerate Search in the Data Cloud Through Generative AI
- Blog: Snowflake acquires Neeva to bring intelligent search to its cloud data management solution
- Deal Value: Approximately $150-185M
- Comment: Founded by former Google executives; founder Sridhar Ramaswamy became Snowflake CEO in February 2024
2023-05-03: Databricks → Okera
- Announcement: Welcome, Okera: Adopting an AI-Centric Approach to Governance
- Blog: A new chapter in our journey: Okera is joining the Databricks family
- Comment: Okera co-founder and CEO Nong Li, creator of Apache Parquet, joined Databricks
2023-02-07: Snowflake → LeapYear Technologies
- Announcement: Snowflake to Acquire LeapYear
- Deal Value: $62M
- Comment: Differential privacy platform to enhance data clean room capabilities
2023-02-08: dbt → MetricFlow (Transform.co)
- Announcement: dbt Labs Signs Definitive Agreement to Acquire Transform, Accelerating development of the dbt Semantic Layer (updated by dbt)
- Press: Dbt Labs acquires Transform, adding semantic tools to its data analytics platform
- GitHub MetricFlow: The Future of the MetricFlow Project
2023-01-19: Snowflake → SnowConvert (from Mobilize.Net)
- Announcement: Snowflake Announces Intent to Acquire Mobilize.Net’s SnowConvert
- Deal Value: 76.3M (approximately 11.6M net of cash acquired)
2023-01-06: Confluent → Immerok
- Announcement: Confluent Announces Intent to Acquire Immerok to Accelerate Development of Cloud Native Apache Flink
- Deal Value: Approximately $54.9M
- Comment: Founded by original creators of Apache Flink
2023-01-05: Qlik → Talend
- Announcement: Qlik Acquires Talend
- Comment: Both companies owned by Thoma Bravo; Talend was acquired by Thoma Bravo for $2.4B in 2021

# 2022

2022-08: Snowflake → Applica
- Blog: Snowflake blog
- Comment: Specializes in decoding and automating complex documents based on deep learning
2022-06-27: IBM → Databand.ai
- Announcement: IBM Aims to Capture Growing Market Opportunity for Data Observability with Databand.ai Acquisition
- Deal Value: Estimated ~$150M
- Comment: Data observability platform; validated data observability as critical category in modern data stack
2022-04-07: NetApp → Instaclustr
- Announcement: NetApp acquires Instaclustr to deliver open source databases as a service
- Comment: Managed open source databases as a service (PostgreSQL, Kafka, OpenSearch, Cassandra)
2022-04: Talend → Gamma Soft
- Comment: Cloud-based data discovery platform based in France, founded 1983
2022-03-02: Snowflake → Streamlit
- Announcement: Snowflake acquires Streamlit for $800M
- Deal Value: $800M
- Comment: Popular open source project for building data-based applications
2022-01-06: Alteryx → Trifacta
- Announcement: Alteryx Announces Acquisition of Trifacta
- Deal Value: $400M cash + 75M retention pool (RSUs)
- Comment: Cloud-based data preparation platform

# Bundling vs. Unbundling

Bundling vs Unbundling

# Once Called, Software is Eating the World

Once called “Software is eating the world”, now it seems the pendulum is swinging back to more unified and integrated data platforms.

I think the best way for OSS products to survive is to embrace the «Declarative Data Stack» approach, where integration happens with a single configuration file. By integrating with multiple tools, you gain the best of both worlds: a combination of integrated and open-source capabilities, along with end-to-end analytics.

However, making money from open-source is hard, but I still hope that many will pursue this path. When deciding on a tool, I will always pick the open-source one. To me, it builds trust, and because it’s shared as a gift for free to use by anyone, it makes me want to support it more.

Consider the Framework Laptops. They’re fully repairable, with every part replaceable, allowing you to swap out the screen or even the motherboard later on. This is something I want to support. Same with OSS. I believe the strategy shouldn’t be to cash out on OSS, but rather have it as a sign of valuing your customer by giving them a gift.

The money stream should be independent of OSS, if at all possible, so you have a clear distinction, and also don’t confuse them with the company down the road. Much easier said than done, but there are still many great data engineering companies that are doing a great job of exactly that. I hope it stays this way.

What do you think, should the vendor still have an open-source product, or focus on making money so it will sustain itself over time?

# Fivetran + dbt

Fivetran + dbt Merger

# Table Formats Market Updates

See Open Table Formats (Market Updates).

# AI Acquisition

Seperate note worthy acquisition in AI related to data engineering.

2026-08-27: NVIDIA -> HuggingFace ^117aa1
- Announcement: Nvidia agrees to buy Hugging Face for $12.9 billion, The Information reports
2026-04-22: SpaceX -> Cursor AI IDE
- X Announcement: > Cursor has also given SpaceX the right to acquire Cursor later this year for $60 billion or pay $10 billion for our work together. NY Times
- Update 2026-06-16: SpaceX locks in $60 billion Cursor deal to close gap with rivals in AI coding race | Reuters
- Update: 2026-08-14 it’s finished: Cursor Tweet
2026-05-06: Codeium, the makers of the Windsurf AI IDE, forked from VSCode were acquired by OpenAI for $3 billion dollars

# Further Reads

Origin: Data Engineering
References: Earning Money with the Open-Source Model—Making Gifts

Notes Compound, Blogs Compile: When I Update a Note vs. Publish a Post

Fri, 02 Oct 2026 13:52:29 +0000

I like to collect notes over time on my Second Brain, and eventually, when I figure something substantial to share, something I learned, I write a blog post about it.

The second brain helps me to share whatever is in my mind, and the blog helps me to refine.

^512110

Notes are compounding and are long-term updates, whereas blog posts are long-term too, but capture the current time of publishing. I share more about how everything compounds, including knowledge, in Long-Term Strategy for Blogging.

# Notes Compound

E.g. Compounding Note-Taking note on this very site looks like this:

You see the last updates, the creation date (when it was created in my private Obsidian vault), and the published date, when it was first uploaded to this page.

So this note will continue to update on its own whenever I find something new to add, a link, or a thought, like this paragraph I’m writing now. From Second Brain Design Updates which triggered this note to update, to then at some point compile into a full blown article as I have done at Why I Still Blog — and Why the Future of Blogging Is Connected with my ten years of writing in public with many concepts from here, from Future of Blogging, etc. complied into a one time blog. This blog won’t get updated. I might write follow-ups like this The Process of Smart Note-Taking or even The Act and the Outcome of Creation, but I won’t change the old one.

But while these blog posts got published, I updated the notes they link to multiple times. So, the notes get updated over time and might lead to another blog post, or just serve as a separate artifact to read and learn in a more connected way.

# Notes and Essays: Compiler

If you ever thought about ’notes vs. a newsletter’, or a ’tweet vs. a blog post’, I think this note is for you, then in my opinion, Notes Compound, Blogs Compile.

The tweets are obviously not the notes I mean, as these are only brief artifacts gone shortly after, but notes in your note-taking app of choice, and when to compile an article from these.
I like how No Boilerplate YT explains how notes compiler to essays (blog).

Source: What do you do after taking notes? with the Zettelkasten method, which I use as well

These can also result in other Types of Writings, such as a book, other Types of Content Online, such as an abstract, dissertation, case study, etc.

# Shake it until New Patterns Emerge

Note taking, compiling and writing an essay is also very much what Sönke Ahrens said in his book Smart Note Taking, one of my favorite book on note taking (Thanks so much DeepRead to point that out):

The note sequences are the clusters where order emerges from complexity. We extract information from different linear sources and mix it all up and shake it until new patterns emerge. Then, we form these patterns into new linear texts. ^d8aeee

# Paper vs. Digital

In a similar comparison, writing on paper vs. digital has also different meanings and values. See more at Digital vs Paper.

# Further Reads

How To Take Smart Notes, by Sönke Ahrens (Book summary) - DeepRead shared at Tweet

Origin: @ssp.sh on Bluesky
References: Future of Blogging

Compounding Note-Taking

Fri, 02 Oct 2026 13:42:31 +0000

Note-taking is like saving money with interest. You save not only the interests of each year but also interest from interests. The same happens if you start taking notes; ideas will compound. Your notes will not only be worth their years but multiplied. My Tweet

Having the Second Brain with Continuous Notes is providing from the compounding effects. Instead of having worth 5 years notes after 5 years, you have compounded them on top. You add on top of existing notes. You do not need to create them from scratch.

That’s why Ideas come so easily with a second-brain approach. A good example is by David Perell, How to Take Notes like Kendrick Lamar.

# Further Reads

Origin: Thomas Bustos said that in our Podcast with Thomas Bustos - Lets Talk AI

Data Engineering Whitepapers

Fri, 02 Oct 2026 11:28:36 +0000

A curated list of influential whitepapers in the field of data engineering.

# Data Lakehouse

Data Lakehouse combines the best of data warehouses and data lakes into a single architecture.

Lakehouse: A New Generation of Open Platforms that Unify Data Warehousing and Advanced Analytics ^6a7f75
Building a Database on S3 (2008) - Before Open Table Formats

# Distributed Systems & Storage

Foundational papers on distributed systems that power modern data infrastructure at scale.

Google File System (GFS): The Google File System ^fdbf43
Data Engineering Architecture: The Google File System (2003)
MapReduce: MapReduce: Simplified Data Processing on Large Clusters

# Data Warehousing & OLAP

Data warehouses and OLAP systems optimized for analytical queries over large datasets.

Data Warehousing: Dremel: Interactive Analysis of Web-Scale Datasets
Dremel Encoding: Dremel: Interactive Analysis of Web-Scale Datasets (Year 2011) used in Apache Parquet, and Apache Drill (which was seeded into Apache Arrow). ^51fc46
DWA: DWA Whitepapers
Redshift Files: Why TPC Is Not Enough: An Analysis of the Amazon Redshift Fleet ^2fbdb2
OLAP - ClickHouse: Lightning Fast Analytics for Everyone
The Snowflake Elastic Data Warehouse: “The mission was to build an enterprise-ready data warehousing solution for the cloud. The result is the Snowflake Elastic Data Warehouse, or “Snowflake” for short.”
Scuba: Diving into Data at Facebook, a paper about Meta Scuba (2013) ^d3683f
- Followed up with Meta’s Next-generation Realtime Monitoring and Analytics Platform (2022) called Kraken: Historically, Meta Scuba has favored system availability along with speed and freshness of results over data completeness and durability. While these choices allowed Scuba to grow from terabyte scale to petabyte scale and continue onboarding a variety of use cases, they also came at an operational cost of dealing with incomplete data and managing data loss. ^e7bf40

# Processing Engines

Data processing frameworks for batch processing and streaming computation.

Apache Spark: Spark: Cluster Computing with Working Sets
Streaming: The Dataflow Model: A Practical Approach to Balancing Correctness, Latency, and Cost in Massive-Scale, Unbounded, Out-of-Order Data Processing
Vectorized Engine: Everything You Always Wanted to Know About Compiled and Vectorized Queries But Were Afraid to Ask ^db4178
Theseus: A Distributed and Scalable GPU-Accelerated Query Processing Platform Optimized for Efficient Data Movement built by Voltron Data ^932b35
End-to-End Declarative Data Analytics: Co-designing Engines, Interfaces, and Cloud Infrastructure: An early prototype by ETH shows how rethinking the interface between the engine and the cloud platform enables elastic data-dependent parallel execution over data lakes, automatic caching, and opens new research directions for cloud analytics. ^b7bd10
Watermarks: MillWheel: Fault-Tolerant Stream Processing at Internet Scale by Google. MillWheel is a framework for building low-latency data-processing applications that is widely used at Google. Users specify a directed computation graph and application code for individual nodes, and the system manages persistent state and the continuous flow of records, all within the envelope of the framework’s fault-tolerance guarantees ^287948

# DuckDB

DuckDB got his own categories as single-file OLAP database.

MotherDuck: DuckDB in the cloud and in the client - A paper that introduces the 1-5-Tier Architecture. ^9f7136
Morsel-Driven Parallelism: A NUMA-Aware Query Evaluation Framework for the Many-Core Age
Unnesting Arbitrary Queries: Unnest subqueries queries in SQL.
Beyond Quacking: Deep Integration of Language Models and RAG into DuckDB: This paper introduces FlockMTL, an extension for DuckDB that deeply integrates Large Language Models (LLMs) and Retrieval-Augmented Generation (RAG) capabilities into database management systems. See an implementation example
Don’t Hold My Data Hostage – A Case For Client Protocol Redesign: Showcasing how large amount of data from a database to a client program is a surprisingly expensive operation and the difference between the clients. ^caa24f

# SQL & Semantics

All about SQL, the domain-specific language to query databases and more.

Spanner: Becoming a SQL System: Google Spanner evolved from a distributed key-value store into a full SQL database system ^786591A
SQL Has Problems. We Can Fix Them: Pipe Syntax In SQL - Pipe Syntax In SQL ^663863
What Goes Around Comes Around by Michael Stonebraker, Joey Hellerstein: This paper provides a summary of 35 years of data model proposals, grouped into 9 different eras. WIth a proposals of each era, and show that there are only a few basic data modeling ideas, and most have been around a long time. ^37c66e
Semantic Data Modeling, Graph Query, and SQL, Together at Last?: Semantic data models express high-level business concepts and metrics, capturing the business logic needed to query a database correctly by Jeff Shute, Colin Zheng and Romit Kudtarkar. CIDR (2026) (to appear) ^e1b169
- Whitepaper- Semantic Data Modeling, Graph Query, and SQL, Together at Last?
Measures in SQL: SQL has attained widespread adoption, but Business Intelligence tools still use their own higher level languages based upon a multidimensional paradigm. ^4b0617
- More in Measure in SQL
Context Layer Models: A language models that natively manage their own context, implemented by treating the context as a file and allowing the model to make unrestricted updates to this file. paper

# Relational Model

Relational databases organize data into tables with rows and columns, pioneered by Edgar F. Codd.

A Relational Model of Data for Large Shared Data Banks, 1970 by Edgar F. Codd ^8d1ede
C-Store: A Column-oriented DBMS: Organizing data by columns rather than rows. 2005 ^fe90b8
- MonetDB/X100: Hyper-Pipelining Query Execution. The beginning of Vectorized Query Execution by MonetDB ^4e54dd

# NoSQL

NoSQL databases trade relational guarantees for horizontal scalability and flexible schemas.

# Schema Evolution

Schema Evolution addresses how databases handle changes to data structures over time.

1995: A survey of schema versioning issues for database systems by John F. Roddick ^eb6505
2012: Automating the database schema evolution process

# Data Architecture & Governance

Patterns for organizing data assets, governance, and discovery across the enterprise.

# Git for Data

Git for Data is version control concepts applied to Datasets and data pipelines.

Reproducible data science over data lakes: Replayable data pipelines with Bauplan and Nessie ^a13d97
Git for Data Paper by xet-core: Proposes a system that extends Git to efficiently handle terabyte-scale machine learning datasets through content-defined chunking and deduplication. ^2e514f
Building a Correct-by-Design Lakehouse: Bauplan’s paper on Git for Data, and building Lakehouse. It contains Data Contracts, Versioning, and Transactional Pipelines for Humans and Agents
GitLake: Git-for-data for the agentic lakehouse by Bauplan: The system lifts single-table Iceberg snapshots into lakehousewide commits, branches, and merges, letting agents work on isolated branches while humans review and publish changes. Pipelines run on temporary branches and publish through a final merge, so all outputs become visible atomically or none do.

# Data Visualization

Data Visualization in data engineering and BI is so imporant.

M4: A Visualization-Oriented Time Series Data Aggregation: Visual analysis of high-volume time series data is ubiquitous in many industries, including finance, banking, and discrete manufacturing. Done by SAP in Dresden and Volker markl Tum, Berlin. ^d2b91d

Not a Whitepaper, but Related

[Grammar of Graphics]]: The Grammar of Graphics approach, popularized by tools like ggplot2 in R and Vega-light in JavaScript, is considered declarative.

# Database Extensibility & Research

Academic and industry research on database design, extensibility, and human-data interaction.

AI Whitepapers

# Other Lists

Schedule - CMU 15-721 :: Advanced Database Systems (Spring 2023) : see linked Papers to each presentation.
Data Engineer Handbook Whitepapers

Origin: Data Engineering Vault

Writing from the Heart: The Best Writing Takes Hours, not Weeks

Fri, 02 Oct 2026 10:07:59 +0000

Whenever I have some time on vacation, I like to write something about my current interest, without much planning, writing the article within a couple of hours (vs. days and weeks I usually spend).

It’s a challenge, but intersting enough; they seem to strike a chord. Whenever I do that and land on Hacker News (my last two, at least). This also happens today with my latest article about « Finding Flow».

There’s a trigger that makes me sit down and just write it. This time, the trigger was the book about Stolen Focus.

I believe this is because when we write fast, we write from the heart. Sometimes, we are angry, or something bothered us over time. Instead of spending “only” a few hours on it, my brain and my subconscious have usually wrestled with it for many days and months. Maybe plain intuition?

# The Phenomen

I’m really interested in this phenomenon of writing straight from the heart. This act itself is an art of flow. Perhaps I will write about it in the following rant :)

It boils down to:

Writing from the heart is the best—true, honest, and genuine human-to-human communication.

# Struggle and Flow of Writing

I love how Jason Fried said it, he does not call it writing from the heart, but writing from a single flow:

For me, when I struggle with writing, it’s a signal to stop writing, not to keep pushing through the writing

You don’t have to struggle through writing, he says:

What’s the point of struggling? There’s some point of struggling in other things, I think like, certain things we’re struggling through; for me, writing is not worth struggling through, it’s worth finding a flow, because then you can ride that ^d82701

# Further Reads

The Act and the Outcome of Creation with Conscious vs. Subconscious

Origin: Steal like an artist Trilogy (Austin Kleon)
References: Writing from a Scar, Intuition, Flow, Art

Finding Flow (Focus)

Fri, 02 Oct 2026 10:02:37 +0000

I wrote a full article on Finding Flow: Escaping Digital Distractions Through Deep Work and Slow Living.

Very related is also Healthy Food. We all know that we should eat healthy, but mostly it’s more convenient to eat unhealthy, fast food.

@ssp.sh on Bluesky

# Writing: Finding Flow

Love Jason Fried’s take on finding flow:

Writing from The Heart
You don’t have to struggle through writing, he says:
What’s the point of struggling? There’s some point of struggling in other things, I think like, certain things we’re struggling through; for me, writing is not worth struggling through, it’s worth finding a flow, because then you can ride that

Writin in flow for me, is with Vim Motions for Writers.

# Further Reads

Origin: Flow State, Finding Flow: Escaping Digital Distractions Through Deep Work and Slow Living | ssp.sh

Writing is Hard; and Meaningless with no Experience

Fri, 02 Oct 2026 10:00:15 +0000

Everyone who can write has a competitive advantage because not everyone can do it. Tim Ferriss said that when he felt competitive in angel investing or podcasting, he was thinking of going back to writing.

In podcasting, with money, you can bring in more money and better production quality. Writing on the other side, is hard. Harder to replicate.

After all, good writing is hard; exceptional writing may be the hardest there is.

# Is AI Solving This?

Big ideas are less about creativity and more about conviction. [..] So, what happened? ‘Sauce and other shit’ got incredibly cheap! [..] There is no AI prompt for conviction. Harry Dry

True writing is hard—and always will be. To me, you are writing from the heart and soul, and that’s a beautiful thing. More so these AI days. Writing from The Heart.

Will AI replace Humans? I don’t think so. There’s no character, no soul, and it’s just average. Sure, it’s good at writing, but it’s not good at having a style, writing new things, or having a hugely large context (e.g., when I write an article over two years long, putting in my heart and soul). These writings will always have their place. We have digital print for a long time, yet books are still a thing.

As Harry says, AI has no conviction; There’s something missing, something human. Or picture this: who would like to read a book and one they liked momentarily, but then you want more and you find out that it’s written by a machine? How would you feel?

# Writing is a Wicked Problem

Writing is a Wicked problem. The Safest Job from AI may be Writing

Love it. To me, writing is also an art form, a craftsmanship, and only with experience do the written words mean something. Everything is meaningless if it’s just for the sake of producing text; if there’s no story, no hardship.

Also:
Large Language Models (LLMs)

# Further Reads

Origin: Tim Ferriss
References: Writing, AI Writing

What I Learned throughout the Last Years Writing with AI

Fri, 02 Oct 2026 10:00:15 +0000

I’ve been writing long before AI, starting in 2015 to publish on this very blog, and then AI came, and I still write manually. Through my experience as a full-time author, these are some of the learnings I got from working and writing with AI over the last 2-3 years. ^bfb3f9

# Learnings

Four things I learned throughout the last years working with AI

If consumed by a human, it should also be written by one

It’s hard to finish an idea that’s not yours

If you start writing today, there’s no way to know if you can write without AI

AI won’t replace human writing (as writing is thinking) ^9893f3

# If Consumed by a Human, it also Should Be Written by One

If Consumed by a Human, Should Be Written by Human
I like Naval’s take, as elaborated above, which says it all depends on who the customer is along the lines of writings are for humans to read mostly, so they should be written by a human.

Code is mostly a tool to achieve something, read by a computer, so it’s less relevant to write by hand, unless you work in a team or need strong maintainability. Full conversations says:

Also, I pondered on the question of Should I Change My Writing because of AI?

# Hard to Finish an Idea That’s Not Yours

Its Hard to Finish an Idea that is not Yours (and suggested by AI)
While I use Claude Code a lot for my Omarchy plugins and Linux optimization/failure detection, I don’t use it for writing and note-taking, especially for coming up with unique ideas and driving them home by writing them through to the end.

Its Hard to Finish an Idea that is not Yours (and suggested by AI)
It’s so hard to finish an idea that is not yours and is just suggested by AI. Instead, I like to Write Manually, Daydream, and follow my own train of thought, even if not all-encompassing, but at least it’s my thought, the idea that has an interest and connection to me.

Related: When Writing in Obsidian, Keep AI out of it

See Keep AI Out of Your Obsidian Vault.

# If You Start Writing Today, There’s no way to Know if You Can Write without AI

Hannes Mühleisen said that he has the benefit of having coded or created DuckDB before the AI hype, so people trust the product, but it’s hard if you start today, it’s really hard to see if a product is good or bad- they all look amazing.

It’s the same for writing. If you can show you have written before using AI, people can trust you more easily than when you start today. If you haven’t, there’s no way for readers to know if you can write without AI, whereas if you have written long enough, people know.

# AI Won’t Replace Human Writing (And Thinking)

As Writing is Thinking, to me, it won’t replace human thinking anytime soon when it comes to writing. No AI can write from experience, from the heart, so even if the writing might be good, it’s soulless, and there’s nothing more. It’s just a soup of words.

I wrote a full essay on Will AI Replace Human Thinking.

# There’s Writing for Humans, and Others for Machines

Joe Reis commented that “there’s writing for humans and writing for machines. These are totally separate audiences”. This is so true. Writing documentation and how-to’s vs. stories.

But still, documentation that is written for humans, for example, and is nicely storied, is so much nicer to read. Also another observation:

If it’s good for humans (CLIs, concise, to the point), it’s probably also good for agents.

And there’s also the factor of time. But in my opinion, everyone just needs to know the consequences of letting it be written by AI (different intent, to elaborate, etc.). A clear button or sign, whether it’s AI-written or not, would help big time.

# Other Learnings: When Writing Code

Origin: AI Writing

Creative Writing

Fri, 02 Oct 2026 10:00:15 +0000

My passion is to inspire people through writing. Most of it comes from being creative, but also productive, Creativity vs Productivity.

# What is Creative Writing?

To me, it’s when you read something, and it just clicks. Written beautifully, removed to the bare minimum, leaving the message there and providing the most value for readers.

# How do I Write Creatively?

Isn’t it the opposite of Writing Effectively? The environment you are in is a big one, as I can’t be creative in a place I always do routine work. I need to do sports, eat healthy, and, most importantly, sleep enough to be fully creative. As well as I need to have done most of my chores. I can’t be creative if I always have in the back of my mind that I should finish my tax report (after five months..). That’s why being creative to me is also saying no to almost all things, like Hell Yeah or No. Because if I constantly say yes to new things, I need to have the headspace to be creative.

# Coffee Break Rythm

I have written more about my Coffee Break Rhythm.

Originally shared on Finding Flow: Escaping Digital Distractions Through Deep Work and Slow Living | ssp.sh.

# Further Reads

My Writing Process
Writing is Thinking
Writing from The Heart
Creative Engine
Earn Attention with Writing
Productive Procrastination
Not everything I write need to be Content
Writing by Hand is Thinking on Paper
My Creative Daily Routine and My Creative Process

Origin: Writing

Archetypes of Notetakers or Writers

Fri, 02 Oct 2026 08:16:03 +0000

Anne-Laure Le Cunff considers three types of note takers: architects, gardeners and librarians. This is what she means by this.

The Architect. They enjoy planning, designing processes and frameworks, and need a note-taking tool that allows them to easily structure their ideas.
The Gardener. They enjoy exploring, connecting various thoughts together, and need a note-taking tool that allows them to easily grow their ideas.
The Librarian. They enjoy collecting, building a catalogue of resources and need a note-taking tool that allows them to easily retrieve their ideas.
Extra: Tiago Forte adds The Student to the list, tough I think they are included in the three above, but maybe they are more exploratory.

More on How to choose the right note-taking app: the ultimate guide - Ness Labs.

Check more on choosing the right tool:

Second Brain
Choosing the Right Tool
It’s important to find the right tool for what you are doing. I like the differentiation between 1. Librarian (mostly stage one when collecting info), 2. Architect (stage two with organizing knowledge), and 3. Gardener (stage three with digesting knowledge and making connections) and choosing the tool accordingly. See Type of Notetakers.

There are 6 type of archetypes in writing as Michael Dean says in How I Write (Podcast).

# Further Reads

Origin: I heard it from Tiago Forte YT video.
References: Archetypes of Stories

My Gigs: Presentations, Podcasts, Videos, Slides, etc.

Fri, 02 Oct 2026 07:10:20 +0000

Here I have a list of my appearances online or in person, with slides I shared during an event.

# 11. First Omarchy Meetup in Switzerland

This was the first Omarchy event in Switzerland that I organized and presented. We talked about all things Omarchy, the cool workflows everyone uses, and plugins people build, and generally about Linux and what OSS can do when the operating system is open. Agents can build on top of it, building the Malleable Computer.

# 10. Knowledge Management in the Digital Age: From Zettelkasten to Startup Owner

It was part of Swiss Digital Days 2025 organized amoung Berner Fachhochschule and Stadtbibliothek Biel. I translated the talk and recorded it to upload to YouTube.

The talks summarized my 20+ years of My Note-Taking Path, and how I use Markdown and Obsidian to create a Second Brain to manage my life and business.

# 9. Andreas Kretz Podcast: Why Writing Is Thinking, and What Data Engineers Can Learn from It

In this episode, Andreas Kretz and I discuss about:

Writing as a thinking tool
Recognizing the same data patterns reinvented over decades
Building a Second Brain to grow and organize your own expertise
Why the real value often lies beneath the tools we use

🎧 Listen as Podcast
📺 Watch on YouTube

# 8. Event: SQL Meets GenBI: AI-Powered Dashboards in a Declarative World

Applied AI, Data & Analytics Meetup at Google Zurich that brought together the worlds of Data Engineering, analytics and AI on June 3rd 2025.
This meetup blends the foundations of modern data infrastructure, analytics with the emerging wave of AI-assisted workflows. Whether you’re building scalable pipelines, experimenting with LLMs for analytics, or creating tools to bridge both—this event is for you!

Meetup with all Talks.
My Talk: SQL Meets GenBI: AI-Powered Dashboards in a Declarative World (internal)
📈 Slides PDF
🌐 Web (HTML) Version
📺 Watch on YouTube

Key Insights

I talk about that while AI handles technical tasks, human judgment and context remain crucial for steering and quality assurance in the GenBI workflow:

GenBI Revolution: After 20 years of unfulfilled promises, AI finally enables true self-serve BI (?) by combining natural language with analytics workflows, moving from linear manual development to iterative human-AI collaboration.

Three Critical Components: Quality data (DWH/organized lake), speed (OLAP systems like DuckDB/MotherDuck), and BI-as-Code (declarative YAML/SQL) form the foundation for effective GenBI implementation.

Declarative as AI Language: LLMs work best with declarative interfaces using YAML for configuration and SQL for business logic - creating a common language between humans and AI that provides essential business context.

Developer Experience Transformation: GenBI reduces time-to-insight from days to minutes, enabling both technical developers and business users to generate dashboards in seconds while maintaining the benefits of version control and automation.

The ChatGPT Moment for BI: Just as ChatGPT demonstrated complex question-answering with minimal effort, GenBI solves the self-serve BI challenge through semantic business layers and MCP integrations that provide contextual understanding.

Self-Serve BI Finally Achieved: After two decades of evolution from pixel-perfect reports to drag-and-drop tools to modern data stacks, 2025 marks the first time true self-service BI is actually possible through AI-powered workflows.

# 7. Joe Reis Podcast: The Art of Writing about Data Engineering

Joe Reis introduction:

Ever wondered how to turn complex data engineering concepts into compelling content? Join me as I dive deep with Simon Späti, a master of writing about data engineering. We explore ‘The Art of Writing about Data Engineering,’ uncovering his journey from business intelligence to becoming a leading voice in the field.

In this episode, Joe and I discuss about:

The essential skills for writing effective data engineering content.
How AI is transforming the writing process (and what it means for you).
My personal journey and the lessons I learned along the way.
The future of technical writing.
How to build an audience through your writing.

🎧 Listen on Spotify
📺 Watch on YouTube

# 6. Data Talks on the Rocks (Rill)

I had the pleasure of talking with Mike Driscoll from Rill Data. Our topics include

My journey from data engineer to author
The hype behind Bluesky🦋
Interesting use cases for DuckDB in production
What do you call data people? analysts? data developers? data engineers?
Latest trends in object storage, data modeling, & declarative stacks

📺 Watch on YouTube
🎧 Transcript

Key Insights

Data Roles: The definition of data professionals (analyst, engineer, etc.) varies significantly by region and company, with many starting in dashboards before moving deeper into the stack.

Bluesky: Gaining popularity due to its open architecture and Twitter-like feel, running on-premise with just 20 people serving 20M+ users using SQLite databases.

DuckDB: Functions as a fast, lightweight OLAP solution excelling as a zero-copy layer, with notable production uses at Okta, Spare Course, and Hugging Face.

Data Stack Evolution: Trending toward object storage (S3) as foundation, with movement from unbundling to rebundling of tools, and growing adoption of declarative approaches.

Data Modeling: Remains crucial despite “schema-on-read” trends, with emphasis on centralized metrics definitions and business rules, though cloud vendor incentives don’t align with efficient modeling.

Future Outlook: Growing importance of human judgment alongside AI/GenBI tools, with a continued need for balancing quick wins against thoughtful architecture.

Your Journey: Discussed my path from computer science apprentice in Switzerland to data engineer, technical writer, and now author focusing on data engineering patterns.

# 5. Bedag Podcast: Open Source Meets Data Engineering

This podcast episode Matthias Stürmer and I dive into the topic of Open Source and Data. Matthias is a professor at the Bern University of Applied Sciences (BFH) and Head of the Institute for Public Sector Transformation. I’m grateful for his time and the insightful discussion we had.

🎧 Listen on Spotify - 🚨 Attention: This podcast is in Swiss German

Key Insights

Open Source: The Core of Innovation

What is Open Source? A behind-the-scenes look: Open Source stands for access and freedom. Unlike Closed Source software like CRM or ERP, Open Source offers the opportunity to customize and further develop the code.

The Open Model: Why it represents the perfect balance between Open Source flexibility and commercial success. No lock-in, more flexibility, and innovation, but with the awareness that more work and maintenance are required.

Open Core or Closed Source? The choice between Buy vs. Build depends heavily on technical capabilities and long-term goals. A Managed Service supporting the Open Core model can be an attractive solution.

Data Engineering: The Future of Data Processing

HelloDATA: An open enterprise data platform, developed by Bedag and adopted by the Canton of Bern. We discuss the advantages of choosing Open Source for data platforms and why it can be a superior alternative to solutions like Azure/Power BI.

Why not Azure/Power BI? A critical look at the lock-in risks and the importance of Open Standards and Open Source for a sustainable data strategy.

Podcast Bedag - OSS Data Engineering

# 4. Open-source data engineering, productivity, writing, and philosophy

In this episode, I had the pleasure of talking to Thomas Bustos on “Let’s Talk AI.” We explored topics close to my heart, ranging from open-source data engineering to the nuances of my productivity workflow. We even touched upon the philosophies I’ve embraced through life’s journey. As a Swiss data engineer who transitioned into a writer and author, this episode encapsulates my two decades of experience and my unbridled passion for productivity and knowledge sharing.

📺 Watch on YouTube
🎧 Listen on Spotify
🕐: 2023-11-28

Key Insights

Reflecting on the Modern Data Stack: The future of data engineering, I see a landscape where tools consolidate, integrating more seamlessly, with Kubernetes steering the ship. My fascination lies in the continuous evolution of the Semantic Layer and the untapped potential of Rust in the realm of data engineering. These advancements might redefine how we approach data engineering in the future.

Career Insights: I advocate for a “pathless path” approach, one that’s guided by curiosity and intuition. Engage in personal projects; they are fun and offer deep, practical learning. Remember, learning is a journey, not a race, so take your time to absorb and understand.

Merging Programming, Writing, and Note-Taking: In pursuing productivity, I’ve found a unique harmony in combining my love for programming with my passion for writing and note-taking. Tools like Obsidian and Vim motions are integral to my workflow, enhancing efficiency and organization. Embracing the Second Brain concept, I’ve created a comprehensive knowledge repository, uniting all these facets seamlessly.

Embracing Minimalism in Work and Writing: My work, both as a data engineer and writer, is positively influenced by minimalism. This approach helps me focus on the essentials, strip away the superfluous, and truly listen to my intuition. This minimalist philosophy aids in making more meaningful decisions in my professional and personal life.

Stoic Principles: Adopting Stoicism has been transformative, helping me manage my thoughts and emotions effectively. The stoic principle of focusing on what I can control and accepting what I can’t has been a guiding light in my work and personal life.

Advice on DE Programming Languages: In the ever-evolving landscape of data engineering, Python and SQL remain the cornerstone, with Rust emerging as a powerful tool for performant applications.

Ending Message: Stay curious, trust yourself, and embrace the journey. I’m a proponent of “Slow Productivity” – a mindful, deliberate approach to work and life.

# 3. TechTalk with Databricks Denny Lee: Data Integration with Airbyte and Delta Lake

In this engaging session with Denny Lee from Databricks, I explored the intricate world of data integration using Airbyte and Delta Lake. We delved into the challenges and solutions of integrating diverse data sources into a unified data lake. This discussion was particularly enriching as it combined practical insights with theoretical understanding, bridging the gap between concept and application in the realm of data engineering.

📺 Watch on YouTube
🎧 Listen on Spotify

Key Insights

Integration Challenges: We discussed the complexity of integrating multiple data sources, emphasizing the importance of robust and scalable solutions like Airbyte for efficient data ingestion.

The Power of Delta Lake: The conversation highlighted Delta Lake’s role in simplifying data management, offering a structured yet flexible approach to handling vast datasets.

Open Source Synergy: A significant focus was on the synergy between open-source technologies like Airbyte and Delta Lake, showcasing how they empower data engineers to build more dynamic and resilient data pipelines.

Future of Data Engineering: The discussion ventured into the future trends in data engineering, touching on the evolving landscape and the role of emerging technologies in shaping it.

# Personal Reflection

The interaction with Denny not only allowed for a deep dive into technical nuances but also offered a chance to reflect on the broader implications and future direction of data engineering. It served as a reminder of the continuous learning and adaptation required in our ever-evolving field.

Date: 2022-12-06

# 2. Data Engineer Turns His Hobby Into a Career

In this heartening podcast episode, I delved into my personal journey from a business intelligence engineer to a data engineer and eventually embraced my passion for writing as a career. The pivot from a conventional BI engineering role to the more dynamic field of data engineering was driven by a desire for automation and a deeper engagement with coding, moving away from the drag-and-drop tools of traditional BI platforms.

📺 Watch on YouTube

Key Insights

Transition from BI to Data Engineering: This episode highlighted my shift from business intelligence, where I worked with tools like SSIS and Oracle, to embracing the challenges and creativity of data engineering. It underscored the evolution of the field and how I adapted to these changes.

Embracing Writing as a Career: Discussing my side hobby of writing about data, productivity, and personal knowledge management, I shared how this hobby transformed into a professional pursuit. This shift was marked by a transition from writing as a personal interest to taking up technical writing as part of my professional role.

The Power of Continuous Learning: The conversation also focused on the importance of continuous learning and staying updated with the latest trends in technology. This aspect has been crucial in both my technical and writing endeavors.

Innovation and Automation: I shared insights into my drive for innovation and automation, a core aspect of my work in data engineering. This reflects my approach to work, where I constantly seek to streamline processes and enhance efficiency.

# Personal Reflection

Reflecting on my journey, this podcast with Chris was a testament to the power of pursuing one’s passions and interests. My story illustrates that career paths are not linear and that embracing change and new opportunities can lead to fulfilling and dynamic career trajectories. This episode serves as an encouragement to others to explore and integrate their hobbies and passions into their professional lives.

Date: 2022-08-23

# 1. Dagster at Rohde and Schwarz Mobile Network Testing

In this podcast episode, I shared the innovative use of Dagster at Rohde & Schwarz Mobile Network Testing. My focus was on detailing the transformation in data architecture at Rohde & Schwarz, showcasing how we harnessed the power of open-source data engineering tools, orchestrated with Dagster.

📺 Watch on YouTube
📄 View Slides

# Personal Reflection

This presentation was a deep dive into how data engineering and architecture can be revolutionized in a corporate setting. It emphasized the importance of embracing open-source tools and technologies to stay ahead in the rapidly evolving field of data engineering. My journey with Dagster at Rohde & Schwarz serves as a case study for others in the industry, demonstrating how adapting to new tools and methodologies can lead to significant improvements in data processing and analysis.

Key Insights

Rohde & Schwarz’s Journey: I described Rohde & Schwarz’s journey in developing smart analytics for mobile network testing, highlighting the company’s focus on electronic test equipment, broadcast and media, cybersecurity, and radio communications.

Innovative Data Architecture: The core of my talk revolved around the innovative data architecture we implemented at Rohde & Schwarz. This included the transition of our data warehouse to the cloud, aiming for big data readiness and the integration of various open-source tools.

Dagster’s Role: I explained how Dagster played a pivotal role in managing and orchestrating our ETL (Extract, Transform, Load) logic in a centralized manner. The move to Dagster was motivated by the need for a more scalable, cloud-compatible tool that could handle increasing data volumes due to 5G advancements.

Custom ETL and Challenges: The custom ETL processes were a significant part of our architecture, dealing with complex data transformation and aggregation. Challenges included handling large data sets, ensuring fast query responses, and maintaining an efficient web UI for user interactions.

Dagster Advantages: I elaborated on the advantages of using Dagster, such as its out-of-the-box features like restart capabilities, backfilling, dependency management, and a user-friendly UI. Dagster’s functional design and the reusability of code significantly streamlined our development process.

Future Directions: Looking forward, I highlighted our plans to enhance our use of Dagster with unit testing, smoke tests, more intensive use of assets, integration of data lineage features, and the exploration of dynamic orchestration and partitioning.

Date: 2021-05-11

Origin: CV
References: Data Engineering Podcasts, Joe Reis Podcast with Me

SecondBrain Disclaimer

Thu, 01 Oct 2026 13:03:06 +0000

A few words on how this public second brain is set up. Be aware that these are essentially my notes shared publicly. So expect incompleteness, grammatical errors, or sometimes broken links, etc.

You can check the status, I call it Taxonomy on each note from started -> growing -> evergreen. Hover the status to see recent git changes.

While not everything is perfect, I also tend not to share useless notes. As these are only my public notes, I have 1000 more notes that are private. Before sharing any note, I evaluate whether it’s useful to someone.

Ultimately, there are notes that might be highly refined, and redacted many times; there might be some ideas quickly jotted down on the go. I have Taxonomy of note types in place, which I sometimes use to categorize notes, but I never managed to update all tags when updating them. But it might still be helpful when you see an evergreen tag, which usually tends to be better quality.

# How links work [internal|public]

Please note that the non-clickable, slightly faded links (red arrow) are internal notes and have not been shared publicly yet. So don’t worry if they don’t work.

Internal working links to public notes (dark arrow) appear underlined while non-public notes (blue arrow) are have a dotted line. External links are shown with pink underline (see pink arrow).

Internal links and can be hovered over to view a short pop up with it’s content and date:

Origin: _index

Digital Garden

Thu, 01 Oct 2026 11:50:14 +0000

A digital garden (sometimes also called a Digital Oasis) breaks away from the traditional structure of a “personal blog” or a glossary, which is mainly linear in nature. Unlike these platforms, where content is presented in a series from an index page, a digital garden offers a multi-dimensional exploration. Here, you can delve into the garden, moving inward to discover various interlinked and sometimes unfinished notes, which I refer to as Continuous Notes. These are dynamic, continuously updated with fresh insights, representing a novel category that I believe is the Future of Blogging; see also the different Types of Content Online.

In essence, a digital garden is a compendium of evolving ideas, unconstrained by publication dates. Exploring connected notes through contextual associations, primarily backlinks, and visualized as graph view. Platforms like Roam and Obsidian exemplify this concept, embodying interconnected note-taking or what’s known as Second Brain. This approach, where you engage in Compounding Note-Taking, amplifies the value of each note.

These notes are less about rigidity or perfection and more about ongoing evolution and personal expression. My digital garden serves as my personal notes repository, marked with #publish for public viewing. This low-friction approach means the content can range from polished pieces to raw, unstructured thoughts.

Essentially, the Digital Garden is a public Second Brain: a collection of unrefined, loosely organized notes on personal websites that continuously develop as a personal knowledge base.

Source by Maggie Appleton ^91b565

# Types of Notes

The notes in a digital garden vary in their level of refinement. At one end of the spectrum, we have Evergreen Note or Permanent Note, representing polished, long-standing content destined for my Zettelkasten. Preceding these are the more transient Fleeting Notes and Literature Notes.

“My Taxonomy”

I use Permanent Notes and Evergreen Notes interchangeably. They form the backbone of my note organization system. My Taxonomy of note types provides an overview of how I categorize different levels of notes.

# Example of Digital Gardens

For a deeper understanding, I recommend the article A Brief History & Ethos of the Digital Garden.

Another notable example is Personal Wiki / Digital Garden, which offers extensive information on Personal Knowledge Management (PKM). This site uses a Wiki format, differing from the Linking Your Thinking (LYT) method by Nick Milo. Networked Thought also presents interesting insights.

Discover more through this curated collection by Maggie Appleton. Also, explore Other Public Second Brains for additional examples.

# Further Reads

See my evolution of Obsidian Graphs from 2021 to today, and how the garden grew over the years. As well as My Note-Taking Path.

Origin: A Brief History & Ethos of the Digital Garden

Four Modes of Writing: Writing with Vim Motions

Thu, 01 Oct 2026 10:32:46 +0000

My writing process has four modes.

NORMAL mode: jumping around, reading, learning
INSERT mode: writing, thinking, making connections
VISUAL mode: copying, highlighting, formatting, designing
COMMAND mode: automate, fix, macros.

In case you can’t tell, I use Vim Motions in Obsidian 😉.

The amazing part is that when you use vim motion long enough, your hands are just doing things. You don’t think “I want to make a list, or I want to write a paragraph, or anything else”. You just think about what you want to write or do, and your fingers just instinctively switch between modes.

See more about how I these in Vim Motions for Writers.

# Futher Reads

Origin: This was inspired by Nick Milo and his four modes
References: My Writing Process

Vim Motions for Writers

Thu, 01 Oct 2026 10:32:35 +0000

Vim Motions for writing are the best flow I can imagine. The smoothness and clarity are on another level.

If you want to see what this looks like, below is a live capture of me writing an article based on my existing notes and navigating my Obsidian vault with Vim motions. The video is sped up 2000% from the original screen capture, reducing 43 minutes to 2 minutes.

Video link to Vim Motion for Writers Showcase (Inside My Obsidian Vault) | This is the publish article shown in the video: Simplicity of a Database, but the Speed of a Cache: OLAP Caches for DuckDB

This is a video I wanted to make for a long time, and today I just took a screen capture while working on the new article (Caching with DuckDB, stay tuned). It showcases the process of writing with vim motions in my Obsidian vault. Until you see it in action, it’s hard to understand. I hope this helps you open a new world of editing text compared to what you might be used to.

However, isn’t it beautiful? Almost like a painter paints their art, or editing words with the precision of a surgeon operating on their patients.

I suggest everyone try it wheter you are a writer or a programmer.

# The Flow of Writing an Essay

This is Paul Graham video on editing with vim, which inspired my captured video above. Here, Paul is showing his vim editing skills. He says Writing is rewriting as an example of his personal Editing and Proofreading.

When you first write part of an essay it tends to flow very well. Then you notice things you overlooked, and you have to go back and insert them. This usually ruins the flow. Fixing this — recreating the old flow with the new ideas — is one of the hardest parts of writing. Paul Graham ( x.com)

How it looks when Paul writes:

What it looks like writing an essay. I use vim and run the text through fmt. Instead of a single wallpaper, I use individual images for decoration. Sometimes this yields interesting combinations. x.com

# Paul Graham Timelapse Video

Writing is rewriting. Here’s me writing the essay “Startups in 13 Sentences.” Characters in yellow will ultimately be deleted. Notice how few characters I type make it into the final version. x.com

Time-lapse: Watch Me Make Mistakes
The finished: Startups in 13 Sentences

# My Writing Process

I write all my blogs, notes and also my book in Markdown with Obsidian and vim motions. If you are not familiar, it’s very hard to know the advantage or how it works, that’s why I took a screencast of me writing.

To me, it’s the best way of editing text, and therefore writing. The best part is being in the flow, moving around without overthinking; the fingers just do the work. I don’t think I could get that flow otherwise, except by writing from start to finish. But that’s not typically how I write.

I start with an outline, add to it over the week and potentially years, and then, at some point, finish it. Changing the re-structure, the flow many times. Truly editing it, where I see vim motions (not the editor) really shine. Read more on My Writing Process.

# Four Different Modes for Writing

I have Four Modes of Writing. From Normal, Insert, Visual to Command mode. Each has their unique way of editing text.

NORMAL mode: jumping around, reading, learning

INSERT mode: writing, thinking, making connections ^e0e158

VISUAL mode: copying, highlighting, formatting, designing

COMMAND mode: automate, fix, macros.

In case you can’t tell, I use Vim Motions 😉.

# Take Out (Overflow), Alternative Titles & Zen Mode

Seeing Jason Fried writing tools that uses Overflow and alternative titles, words and paragraphs, made me write up my own writing flow, that is very similar.

I started to use # Take Out header at the very bottom of each of my article, and I use vim - so G will jump to the end, I paste it, or put take outs there, ctrl-o jumps back where I was. If I want them side by side, in Obsidian I open the same Markdown twice, and one is at the place where I write.

I also use alternative titles in the header as lists. And for alternative words, I use the good old /, but I never thought about hiding it. Very neat, I will think about more. This is how it looks:

As you can see, I also use a shortcut to strike-through bullets and notes, ideas that I have integrated, so I know what I could use still, similar to ghosting it out that Jason did in his showcase.

I also use Zen Mode for Writing.

# Dedicated Devices for Writing

Besides the way I write, I also have dedicated Uni-taskers device to write, like the old typewriters, I have one inherited from my grand dad, a My Typewriter (Hermes 2000), but I have also a new digital version of it, My Distraction-Free Typewriter (Micro Journal).

Both of them I use for different tasks, and in different situations. But the most fun is exchanging them.

See more Typewriters or Desksetups

Today’s Office or Computer Desk Setup and Monitor Workflow.

# Futher Reads

Origin: Vim Language (and Motions)

Format USB Drive on Linux

Thu, 01 Oct 2026 09:55:35 +0000

Check with lsblk what drive it is, usually sda:

sudo umount /dev/sda1
sudo wipefs -a /dev/sda
sudo parted /dev/sda --script mklabel msdos mkpart primary 1MiB 100%
sudo mkfs.exfat -n TUXEDO /dev/sda1

# Helper Script

I have a helper script at usb-format that can do it in one line format and label for USB drive - examples:

# Usage: usb-format <device> <fat32|exfat|ext4> <label>
usb-format sda fat32 TUXEDO
usb-format sda exfat TUXEDO
usb-format sda ext4 TUXEDO

There’s some safety included to not format OS or other partitions, only USB.

Origin: Omarchy

Omarchy: An Opinionated Arch/Hyprland Setup

Thu, 01 Oct 2026 07:24:25 +0000

Omarchy turns a fresh Arch Linux installation into a fully-configured, beautiful, and modern web development system based on Hyprland by running a single command. Created by DHH and hosted on GitHub.

That’s the one-line pitch for Omarchy (like it was for Omakub). No need to write bespoke configs for every essential tool just to get started or to be up on all the latest command-line tools.

Omarchy is now a distro, as DHH would say:

It’s a Linux distribution built on top of Arch Linux that uses Hyprland. Like Ubuntu is a Linux distribution built on top of Debian that uses Gnome.

Find more information on their Website and very well-written Omarchy Manual. Find the phylosophy Omakase behing Omakub, Omarchy at Omakase Computing, or DHH’s Linux Doctrine. Omakase is a mise en place style that chefs use to prepare their food, but for computer developers.

As of today (2026-08-21), never would I have thought that Linux would be the better OS than, let’s say, MS Windows, and even less so than MacOS, but it’s the best I’ve ever used yet. And it keeps getting better. The last Quattro version is outstanding. Plus fun!

Really looking forward to what the next years of Linux will bring.

# My Setups

Find more of them on Omarchy Workstations (Official), or Omarchy Workstation Setups (Deepakness).

# Features

# Backup and Restore Options

Omarchy comes with in-built Snapper backed in that handles system snapshots. It’s integrated into with the bootloader Limine:

Check out more in this PR: Add Limine + Snapper support by ryanrhughes · Pull Request #998 · basecamp/omarchy · GitHub

# Running MS Windows

Running Windows in a Docker VM, check out my PR, hopefully it will be merged soon and available to everyone: Adding seamless Windows VM (with docker) launcher as native app by sspaeti · Pull Request #1333.

# Custom

# Image search

I added an OCR image search with Tesseract.
TUI for image Search

# Themeing All across

Before I had my two-three favorites themes, and always dark mode. But now that switching is so easy, I catch myself changing based on work, mood or sunlight. So it’s actually quite useful, beyond just pure astetics.

This is what all changes with a single theme change:

Terminal colors
Update Nvim theme
Obsidian theme
Browser theme (Brave, Chromium)
btop theme
Background image
Top bar
and more

Short Video on Twitter

All the Oma tools, like Omawrite, Omacalc, etc., also follow along.

# Settings

# Time Delay until Hibernation when Suspending

cat /etc/systemd/sleep.conf.d/hibernate.conf
[Sleep]
HibernateDelaySec=90min
HibernateOnACPower=no

It was set to 30 min, I changed it now to 90 min. this means if I suspend Arch, after 90 min, it will hibernate it (put the data into RAM and fully shut down (no battery used)).

# Printer

Adding Printer on Linux

# My Custom Omarchy Dotfiles

If you are curious, how I set it up, find my dotfiles online, and the Omarchy custom configs live mostly in hypr dots.

# Edge Light for Video Calls

See my small tool: GitHub - sspaeti/wayland-edge-light-videocalls: Edge Light for Video Calls for Linux/Hyprland

# Plugins

Personal Clipboard manager: omarchy-clipboard-plugin: Improved Omarchy Clipboard manager, tailored to my needs · GitHub
Previously in-built: Walker Launcher with Clipboard manager.

# Omarchy Plugin Marketplace

Browse Plugins | Omarchy Plugins

# My Plugins

# Timezones

Timezones | Omarchy Plugins

# Better Clipboard Manager with OCR

GitHub - sspaeti/omarchy-clipboard-plugin: Improved Omarchy Clipboard manager, tailored to my needs · GitHub

# Goatcounter

Website traffic inside your OS.

GoatCounter
I’ve added a Plugin Goatcounter for beautiful website statistic from directly within your operating system Omarchy. This is how it looks with the Gruvbox theme (it automatically adapts to the Omarchy theme):

Check it out at the Market place at Plugin Details | Omarchy Plugins.

# Custom Battery Charge Limit

# Mix Up

# Omarchy Apps

Omapic, is a small little tool to Horizontal and Vertical Cut Out like Snagit does on Hyprland and Omarchy, heavily inspired by omacut.

# Omasnap

Existing screencasting tool with annotations, where I ported the Omapic functionality into it.

Add cut-out tool and configurable preset colors by sspaeti · Pull Request #71 · tobi/omasnap · GitHub

Video here:
Simon Späti 🏔️ on X: “@tobi Here’s a better showcase of how the horizontal cut-out works, built into Omasnap.” / X

# News

# 2026-08-21 Omacon Foundation Starts with 8 Million

Omacom Foundation launches with $8 million — Omarchy News

# Installing it

Download latest ISO from omarchy.org (or curl -o ~/Downloads/ISOs/omarchy-4.0.4.iso.sha256 https://iso.omarchy.org/omarchy-4.0.4.iso.sha256 for version of 2026-10-01)

Use a flash to USB tool, e.g. caligula in the terminal for linux:

`1`	`sudo pacman -S caligula`

plugin USB stick and find it’s path:

 lsblk -o NAME,SIZE,TRAN,MODEL,MOUNTPOINTS
NAME          SIZE TRAN   MODEL                   MOUNTPOINTS
sda          57.8G usb    DataTraveler 3.0
└─sda1       57.8G                                /run/media/sspaeti/KINGSTON
...

sda/1 for my case. then fully umount with above mountpoints so it’s free to flash:

`1`	`umount /run/media/sspaeti/KINGSTON`

Then burn/flash with to USB

`1`	`caligula burn ~/Downloads/ISOs/omarchy-4.0.4.iso -o /dev/sda -z none --hash skip`

Confirm device and put in sudo password for writing if prompted. Then enjoy the view:

And after, replug stick first, confirm lsblk shows sda 57.8G, unmount sda1 if auto-mounted.

# Further Reads

# Troubleshoot Omarchy

Troubleshoot Omarchy

# Linux Specific

(Linux) Philosophy
Efficient Linux at the Command Line
Linux Commands / Essential Linux Commands

Origin: DHH created, follow up of Omakub
References: My Arch Linux for Daily Driver, AUR

Querying Bluesky with DuckDB and SQL

Mon, 28 Sep 2026 16:55:28 +0000

Querying Bluesky directly with DuckDB and SQL is easily possible because the AT Protocol is publicly available and has a Jetstream that you can read.

# Examples

# Engagement Stats

This is the recent Bluesky Engagement Stats ordered by engagement, including a chart:

See origin on Bluesky post

# Query

# DuckDB Obsidian Extension

-- Charting bars with engagement metrics
WITH raw_data AS (
  SELECT * FROM read_json_auto('https://public.api.bsky.app/xrpc/app.bsky.feed.getAuthorFeed?actor=did:plc:edglm4muiyzty2snc55ysuqx&limit=100')
),
unnested_feed AS (
  SELECT unnest(feed) as post_data FROM raw_data
),
engagement_data AS (
  SELECT 
    RIGHT(post_data.post.uri, 13) as post_uri,
    --post_data.post.uri as post_uri,
    post_data.post.author.handle,
    LEFT(post_data.post.record.text, 50) as post_text,
    post_data.post.record.createdAt as created_at,
    (post_data.post.replyCount + 
     post_data.post.repostCount + 
     post_data.post.likeCount + 
     post_data.post.quoteCount) as total_engagement,
      post_data.post.replyCount as replies,
      post_data.post.repostCount as reposts,
      post_data.post.likeCount as likes,
      post_data.post.quoteCount as quotes,
  FROM unnested_feed
)
SELECT 
  post_uri,
  created_at,
  total_engagement,
  bar(total_engagement, 0, 
      (SELECT MAX(total_engagement) FROM engagement_data), 
      30) as engagement_chart,
  replies, reposts, likes, quotes,
  post_text,

FROM engagement_data
WHERE handle = 'ssp.sh'
ORDER BY total_engagement DESC
LIMIT 10;

post_uri	created_at	total_engagement	engagement_chart	replies	reposts	likes	quotes	post_text
3mnct7grlyc25	1780411227949	23	██████████████████████████████	1	2	20	0	If you want to read your recent engagement stats f
3mmnticitu22h	1779689970939	19	████████████████████████▊	2	0	17	0	Touch some grass, they say. 🇨🇭
3mn2s5af4uk2u	1780135202446.637	18	███████████████████████▍	3	1	14	0	The fastest way to learn. Swyx: “Create learning
3mmvm7zvud22z	1779957055219	18	███████████████████████▍	2	0	16	0	Running a DuckDB as a server, setting up from scra
3mn7tvavtrs2l	1780308880778	12	███████████████▋	2	1	8	1	My second brain just got a lot more powerful. Ins
3mnfozcj5ks2q	1780509806399	11	██████████████▎	1	1	7	2	101 concepts every data engineer should know. If
3mmwzgy264t2o	1780005606449.243	10	█████████████	1	1	8	0	Data lakes solve flexibility. Data warehouses solv
3mn7uh2svps2o	1780309478371	10	█████████████	0	1	9	0	This wealth of knowledge is free of charge and upd
3mnf62txilk2i	1780491604591.725	10	█████████████	2	0	8	0	Using DuckDB as a caching layer for sub-second ana
3mnah6tsa632z	1780329603611.699	9	███████████▋	1	0	8	0	Note-taking is like saving money with interest. Yo

done with Obsidian DuckDB Extension

See more at DuckDB with Obsidian.

# Live on Page with Workbench

Run the query live on this page:


-- Charting bars with engagement metrics
WITH raw_data AS (
  SELECT * FROM read_json_auto('https://public.api.bsky.app/xrpc/app.bsky.feed.getAuthorFeed?actor=did:plc:edglm4muiyzty2snc55ysuqx&limit=100')
),
unnested_feed AS (
  SELECT unnest(feed) as post_data FROM raw_data
),
engagement_data AS (
  SELECT 
    RIGHT(post_data.post.uri, 13) as post_uri,
    --post_data.post.uri as post_uri,
    post_data.post.author.handle,
    LEFT(post_data.post.record.text, 50) as post_text,
    post_data.post.record.createdAt as created_at,
    (post_data.post.replyCount + 
     post_data.post.repostCount + 
     post_data.post.likeCount + 
     post_data.post.quoteCount) as total_engagement,
      post_data.post.replyCount as replies,
      post_data.post.repostCount as reposts,
      post_data.post.likeCount as likes,
      post_data.post.quoteCount as quotes,
  FROM unnested_feed
)
SELECT 
  post_uri,
  created_at,
  total_engagement,
  bar(total_engagement, 0, 
      (SELECT MAX(total_engagement) FROM engagement_data), 
      30) as engagement_chart,
  replies, reposts, likes, quotes,
  post_text,

FROM engagement_data
WHERE handle = 'ssp.sh'
ORDER BY total_engagement DESC
LIMIT 30;

# Reading Posts

Bsky and GitHub Gist:

Queried with Query databases in vim (vim-dadbod) within Neovim.

# Query

Or reading the posts - again, run the query live on this page:


-- Query the API directly and flatten the nested JSON structure
WITH raw_data AS (
  SELECT * FROM read_json_auto('https://public.api.bsky.app/xrpc/app.bsky.feed.getAuthorFeed?actor=did:plc:edglm4muiyzty2snc55ysuqx&limit=10')
),
unnested_feed AS (
  SELECT unnest(feed) as post_data FROM raw_data
)
SELECT 
  -- Post basics
  post_data.post.uri as post_uri,
  post_data.post.author.handle as author_handle,
  post_data.post.author.displayName as display_name,
  
  -- Post content
  post_data.post.record.text as post_text,
  post_data.post.record.createdAt as created_at,
  
  -- Engagement metrics
  post_data.post.replyCount as replies,
  post_data.post.repostCount as reposts,
  post_data.post.likeCount as likes,
  post_data.post.quoteCount as quotes,
  
  -- Embedded content (if available)
  CASE 
    WHEN post_data.post.embed IS NOT NULL 
    AND post_data.post.embed['$type'] = 'app.bsky.embed.external#view' 
    THEN post_data.post.embed.external.uri
    ELSE NULL
  END as embedded_link,
  
  CASE 
    WHEN post_data.post.embed IS NOT NULL 
    AND post_data.post.embed['$type'] = 'app.bsky.embed.external#view' 
    THEN post_data.post.embed.external.title
    ELSE NULL
  END as embedded_title,
  
  -- Total engagement score
  (post_data.post.replyCount + 
   post_data.post.repostCount + 
   post_data.post.likeCount + 
   post_data.post.quoteCount) as total_engagement
FROM unnested_feed
ORDER BY created_at DESC;

Uploaded to Reading bsky posts with DuckDB example. · GitHub, too.

# Get Unique ID

Get the Unique ID of my User through public API using HTTPS community extension

D WITH __input AS (
    SELECT
      http_get('https://public.api.bsky.app/xrpc/com.atproto.identity.resolveHandle?handle=ssp.sh') AS res
  )
  SELECT
    res::json->>'body' as identity_json
  FROM __input;
┌────────────────────────────────────────────┐
│               identity_json                │
│                  varchar                   │
├────────────────────────────────────────────┤
│ {"did":"did:plc:edglm4muiyzty2snc55ysuqx"} │
└────────────────────────────────────────────┘

Also find a older Git repo with some queries

GitHub - sspaeti/bsky-atproto: Playground for bsky-atproto · GitHub

# Attach Bluesky Firehose as database managed by Query.Farm

Rusty Conover created a git repo vgi-bluesky, that adds Bluesky as ordinary DuckDB tables: a read-only VGI worker. Posts, threads, profiles, followers, likes, feeds, trends and the Jetstream firehose, over SQL.

Install (if not from community repo):

uvx haybarn-cli
⠇ Preparing packages... (0/1)
haybarn-cli          ------------------------------ 5.17 MiB/21.11 MiB                           
memory H INSTALL vgi FROM community;
memory H LOAD vgi;

Then you can attach bluesky database with:

1
2

memory H ATTACH 'bluesky' (TYPE vgi,
	   LOCATION 'uvx --from git+https://github.com/Query-farm/vgi-bluesky@v0.1.0 vgi-bluesky');

And run queries:

memory H SELECT distinct category FROM bluesky.trends
         ;
┌───────────────┐
│   category    │
│    varchar    │
├───────────────┤
│ politics      │
│ entertainment │
│ science-tech  │
│ sports        │
│ business      │
│ culture       │
└───────────────┘

memory H SELECT rank, display_name, category, post_count FROM bluesky.trends where category='science-tech' ORDER BY rank;
┌───────┬────────────────────────────────────┬──────────────┬────────────┐
│ rank  │            display_name            │   category   │ post_count │
│ int64 │              varchar               │   varchar    │   int64    │
├───────┼────────────────────────────────────┼──────────────┼────────────┤
│     5 │ SpaceX launches Starship Flight 14 │ science-tech │        418 │
│    13 │ Meta launches Muse AI agent        │ science-tech │       3713 │
└───────┴────────────────────────────────────┴──────────────┴────────────┘

# Further Reads

Origin: ATProto

Second Brain: Crafted, Curated, Connected, Compounded on

Claude Dashboards: The Jupyter Notebooks of Agent Age

# What Claude Dashboards Can’t Do: Disadvantages

# Claude Dashboards Are the Jupyter Notebooks of the Agent Age

# BI-as-Code is Still the Future

# Having a Dedicated Tool vs. the All-encompassing

# Example of Claude Dashboard (Compared to BI Tools)

# Dashboards in Rill, Evidence and Claude Code

# The Differences between a BI Tool and Claude Dashboard

# Claude Dashboard Artifact Runs only inside Claude

# Good for PoC or Analysis of Your Data

# Continuing Maintenance: That’s why I Prefer SVG

# Further Reads

Different Types of Writers

# Two Types of Writers

# Writing Process

# What’s Your Choice?

# Different Notes and Notetakers

# Structure of Writings

# FAQ

# Writing vs. Typing

# Outline vs Chain of Thoughts

SQLMesh

# Docs

# Plans (Environments)

# Virtual Environments

# They Have Column-Level Lineage

# Installation on top of dbt

# Semantic Understanding

# Using SQLGlot Expressions

# Integrations

# Features

# History of SQLMesh

# Roadmap after Acquisition

# Tobiko Acquired by Fivetran

# Tobiko Acquires Quary

# Joining Linux Foundation

# Alternatives

# Dbt Vs SQLMesh

Photoshop Alternatives

# Tools

# Free Adobe Copies for Mac, Windows and Linux

# LightCraft

# Lightroom Alternatives

# Further Reads

Listmonk

# Settings

# Automatic Confirm Subscription Email by Clicking in Email

# Welcome Emails

# Opt-In Campain to None-confirmed Subscribers

# Listmonk Analytics (Views + Clicks)

# Automation

# Sending Emails of Newsletter with Amazon SES

# RSS

# This is how it Looks

# Template for Automating My Newsletter with Infos from SB and Bluesky

# Integration into My Email Client

# Interface

# Querying

# Subscribe with Curl

# Tech Setup and Maintenance

# Upgrade Listmonk on Railapp

# Backup Postgres DB

Restoring data from the backup

# Caddy: CORS Settings - Railapp Migration

# Analytics: I Created My Own Dashboard Based of Postgres DB

# Opening Rate

# List and Subscribers

# Listmonk vs. Alternatives

# Listmonk vs. Substack

# Further Reads

BI Tools

# Tools

# BI-as-Code

# Early Open-Source SQL-First Dashboarding

# Enterprise Self-Service Platforms

# Semantic-Layer-First

# Analyst & Spreadsheet Workspaces

# Embedded Analytics & Data Storytelling

# Observability & Search Analytics

# Subscribe with `Curl`