Second Brain: Crafted, Curated, Connected, Compounded on

(HEY) Email Screener

Wed, 09 Sep 2026 09:08:09 +0000

The HEY screener is probably one of the biggest life improvements introduced in the last two years (for productivity) that I can’t live without anymore.

Once you experience it, you can’t go back to a normal email client. It’s horrific whenever I see others’ inboxes 😱. The screener gives you emails you actually want, and only them!

# What’s an email screener?

This was invented by the HEY email client and the 37Signals team, and they explain it like this:

You screen your calls, so why can’t you screen your emails? With HEY, you can. HEY puts you in complete control of who’s allowed to email you. The first time someone emails you, you get to decide if you want to hear from them again.

# Alternative Implementations

Because I couldn’t live without it, but I wanted to have a terminal email client, I had to rebuild it inside Neomutt, and then my own email client:

HEY-Screener in (Neo)Mutt
Neomd: My own TUI email client that I use every day, comes with HEY screener built in.

I’d say blocking in Superhuman or other email clients is technically the same, but not emotionally. Because you will see it and you need to do something with it. If it’s screened out by default, it can’t stay in your subconscious. It’s very powerful. Instead of having an open to-do list that everyone can add to, you get to choose who can add 🙂.

# Further Reads

My essay against Email - before I had the HEY screener: Email, and the way we (should) communicate at work | ssp.sh
Terminal-based Email Clients

Origin: HEY-Screener in (Neo)Mutt

HEY-Screener in (Neo)Mutt

Wed, 09 Sep 2026 08:52:15 +0000

To create my favorite feature from HEY.com, the E-Mail Screener, I had to get the existing work and screen in and out of the list for two years or so. Below is how I made it.

But why I am doing this? I enjoy Neovim and its Vim Language (and Motions), and HEY feels very laggy at times, I wanted to try could build a Screener, with a little help from my AI friend 😉.

And Spoiler alert, it actually works. Next up, I will also add the Feed and Paper Trail list.

If you like the video, you can also check my Asciinema video

Find my very own Email Client

neomd: A minimal terminal email client for people who write in Markdown and live in Neovim. Is based on the Screener features too.

“Getting all ScreenIn and Out emails from Hey.com”

To see how I downloaded all emails from Hey.com, see HEY.com- Get a list of ScreenedIn and Out emails.

Since March 2026 there’s now a TUI for HEY

Check out hey-cli, and my Tweet

# So what is a Screener?

Screen emails like you screen calls. You screen your calls, so why can’t you screen your emails? With HEY, you can. HEY puts you in complete control of who’s allowed to email you. The first time someone emails you, you get to decide if you want to hear from them again.

# How is this different from a spam filter?

HEY also has a spam filter which every email hits before it hits The Screener, but plenty of people/services email you who would never be considered spam. A newsletter someone else signed you up for, a salesperson reaching out to inquire, a friend of a friend’s uncle’s brother’s neighbor who got your email address somehow. The Screener puts you in control of who’s allowed to email you — and who isn’t.

# Use-Cases

ToScreen: None screened emails land in that folder
ScreenedOut: Mail from screened out list will automatically moved out of the inbox to that list
PaperTrail, Feed: The same goes for these two list, emails from that list go automatically into this folder
INBOX: Leftovers are the only emails I actively screened and approved. No email will reach my inbox otherwise.

# How I use it

S: shortcut to sync IMAP with my local state.
A: to automatically run the above logic to move email around according to my lists.
O/I: for screen Out or In a selected email, P for adding to PaperTrail, F for adding to Feed.

Find all shortcuts on GitHub here: dotfiles/mutt/.config/mutt/notmuch at master · sspaeti/dotfiles · GitHub

# Screener Setup

Creating Screening Lists

First, set up your screening lists: screened_in.txt, screened_out.txt, feed.txt, and papertrail.txt. These lists will contain email addresses that you want to categorize.

Script Configuration

Use the provided Bash script to process incoming emails. The script checks the sender of each email against your screening lists and moves the email to the appropriate folder:

Emails from senders in screened_in.txt remain in your inbox.
Emails from senders in screened_out.txt go to the ScreenedOut folder.
Emails from senders in feed.txt are moved to the Feed folder.
Emails from senders in papertrail.txt go to the PaperTrail folder.

Update the mail_dir variable in the script to point to your Neomutt mail directory.

Setting Up NeoMutt Macros

In your .muttrc file, set up the following macros for easy management of your screening lists:

I: Screens in the sender of the current email.
O: Screens out the sender of the current email.
F: Adds the sender to the Feed list.
P: Adds the sender to the PaperTrail list.

These macros automate the process of updating your lists directly from within NeoMutt.

Usage

Run the script periodically (e.g., via a cron job) to process new emails. Use the NeoMutt macros to quickly add or remove senders from your lists, streamlining your email management process.

# Setting up Neomutt Offline

I followed this Setting up neomutt with offlineimap and msmtp, but you can see my Mutt Dotfiles.

# OfflineIMAP

`1`	`brew install offlineimap`

Offlineimap has an error, fixing it manually in /opt/homebrew/Cellar/offlineimap/8.0.0_1/libexec/lib/python3.11/site-packages/offlineimap/LocalStatusSQLite.py according to LocalStatusSQLite.py (fixed in master, but no new release)

# Config

[general]
ui = ttyui
accounts = domain.com

#### account 1 ------------------------------------
[Account domain.com]
localrepository = domain.com-local
remoterepository = domain.com-remote

[Repository domain.com-local]
type = Maildir
localfolders = ~/Documents/mutt/domain.com
status_backend = plain

[Repository domain.com-remote]
type = IMAP
auth_mechanisms = PLAIN, LOGIN
remoteuser = simu@domain.com
remotepass = my pass - escape every % with %%
remotehost = imap.mail.hostpoint.ch
remoteport = 993
ssl = yes
sslcacertfile = /opt/homebrew/etc/openssl@3/cert.pem

# msmtp

I use new iSync (mbsync)

1
2
3

brew install msmtp
vim ~/.msmtprc
chmod 600 ~/.msmtprc

my working config:

defaults
auth on
tls on
tls_starttls on
tls_trust_file /opt/homebrew/etc/openssl@3/cert.pem
syslog LOG_USER
logfile ~/.msmtp.log
timeout 60

#### account 1 ------------------------------------
account domain.com
host asmtp.mail.hostpoint.ch
domain domain.com
port 587
auth on
from simu@domain.com
user simu@domain.com
password my password, don't escape % as above

# Set this account as the default
account default : domain.com

testing with:

`1`	`echo "Subject: Oye from msmtp" \| msmtp test@gmail.com`

# Neomutt

### offline email -------------------------
set sendmail = "/usr/bin/msmtp -a domain.com"
set folder = "~/Documents/mutt/domain.com/"
alias me "Simon" "simu@domain.com"
set mbox_type = Maildir
set ssl_starttls = yes
set ssl_force_tls = ye

# Updated with notmuch

I updated 2025-10-21 with Notmuch and mbsync find most in :

# Resources

Origin:

Everything I've Published Came out of One Obsidian Vault

Wed, 09 Sep 2026 08:40:38 +0000

When checking how much I have written over the years, it amazes me every time, and most of the inspirations, ideas, and brainstorm came initially from my Second Brain. Because I write everything (my notes, blogs and my book) in Markdown, it’s easy to count them, which I did below.

Below you see how notes helped elevate to full blown blog post or chapters of my book plus some numbers and how I published over the years.

# What I Have Published and Written so far in Numbers

Here’s the numbers as of today (2026-08-02):

Page	files	words
Second Brain	8,360	3,066,576
Blog	103	329,934
PoDE Book	105	157,089
Total	8,568	3,553,599

The full break down of files, PDFs, imges and roughly what I have written vs. quoted or web clipped - I try to only count my own as much as possible, you see at the end Full Breakdown of Numbers.

# From Note to Published; the Process

Most blog article and book chapters started as a note I linked to other notes, and only later did the shape of a blog post or chapter I could use for my book.

Below you see some notes I created over the years, and how the lead into blog posts or book chapters. There’s 100 more, but just to give an example.

Note	Became
Semantic Layer	note (2022) → blog The Rise of the Semantic Layer + a three- part series → book chapter Business Intelligence, Semantic Layer, Modern OLAP, Data Virtualization
Zettelkasten	note since 2021, still edited in 2025 → PKM Workflow for a Deeper Life & My Obsidian Note-Taking Workflow - 4+ years of edits, never “finished”, always compounding

And it’s rarely one note becoming one article, it’s more often many notes converging into a single post. Three examples:

Finding Flow grew out of notes like Deep Work, Music that keeps you focused, Productive Procrastination, Social Media Addiction and Journaling and many more.
Why I Still Blog pulled together Writing from the Heart, Learning in Public, Compounding, Future of Blogging and more, see also linked notes in blog.
and the same with book chapters: Materialized View vs. OBT vs. dbt vs. Traditional OLAP vs. DWA grew out of separate notes on Materialized Views, One Big Table, dbt, OLAP and Data Warehouse Automation, each one written years apart, that I only later saw were the same pattern.

# Compounding Effect on Note-Taking

This is essentially the difference between writing and curating a note over time, and compressing it into a single timely blog:

Sharing as Second Brain Note vs a Blog Post
The second brain helps me to share whatever is in my mind, and the blog helps me to refine.

I’ve also written about the compounding effects of note-taking, and how knowledge can grow similar to money, passively, if you start connecting and adding to it. In the end,
everything does compound.

I share the full details and code for the second brain at Public Second Brain with Quartz.

# Shared Words Online and Private

# Full Breakdown of Numbers

As of 2026-08-02 , here’s what’s published and what’s still private in my Second Brain:

                    notes          words
All                 8,360      3,066,576
Published             660        308,421
Private             7,700      2,758,155
Archive               116         75,210
Active              8,244      2,991,366

What I have written vs. quoted (mostly web clipped):

mine   2,284,330
quoted  191,287
clipped 372,787
skipped 205,586

Notes vs. images and PDFs - amazing 9k files and ~3 million words are only 27,4 MB big:

                  files         size
Notes             9,621        27.4M
PDFs                 50        58.5M
Images            7,680         3.0G
Other                14         5.8M
─────
Total            17,365         3.1G

Or overall with my Patterns of Data Engineering book and Blog:

                      files          words
Second Brain          8,361      3,067,475
Book chapters           105        157,089
Blog posts              103        329,934
─────
Total                 8,569      3,554,498

I was curious to see how many books that equate, Books Words and Pages they say a typical range is typical range is between ~125,000-150,000 words. That would be ~23 books I’ve written.

For sure, there are notes and things I have copied, or highlighted, or some other text I didn’t exclude, though I tried to exclude everything I know I added, such as my bible inside my second brain, that I haven’t written. But even if I only count half of it, let’s say 11-12 books, it’s still massive if you think about it.

# My PoDE Book Progress

To document the progress of my book writing, I created a little script that outputs words written over time.

Markdown files in my Obsidian Vault:

`1`	`find ~/Simon/Sync/SecondBrain -type f -name "*.md" \| wc -l`

or another one:

`1`	`find src -type f -not -path '/\.' -name '*.md' -exec cat {} \; \| wc -w`

Explanation:

find /path/to/your/obsidian/vault: Searches the specified directory.
-type f: Ensures that only files are considered (not directories).
-name "*.md": Filters the search to only include files with the .md extension.
| wc -l: Counts the number of lines output by the find command, which corresponds to the number of Markdown files.

# Data Views

`1`	`$=dv.pages().length`

# Over time with git (DEDP book)

Interesting is to count the words over time. As I have everything in git, I loop over git logs with this script words-history.sh:

#!/bin/bash

last_count=0

# Iterate over each commit
git log --reverse --pretty=format:"%H %ad" --date=short | while read hash date; do
    # Check out the commit
    git checkout $hash &> /dev/null

    # Count the words in markdown files
    words=$(find src -type f -not -path '*/\.*' -name '*.md' -exec cat {} + | wc -w)
    
    # Compare with the last count and output only if different
    if [ "$words" -ne "$last_count" ]; then
        echo "$date,$words"
        last_count=$words
    fi
done

# Check out the original branch
git checkout main

Now I can run below in my Makefile:

`1`	`./words-history.sh > stats/stats-words.csv`

With Rill Developer, I imported and created CSV and made a dashboard with

1
2

rill init
rill start

Simple is beauty.

# Further Reads

How to Take Notes in 2021? | ssp.sh: From 10 years of OneNote to Obsidian
My Obsidian Note-Taking Workflow | ssp.sh
The Process of Smart Note-Taking | ssp.sh
and some behind the scence of how I write in Why I Still Blog — and Why the Future of Blogging Is Connected | ssp.sh
Obsidian Search (lots of files)
Books Words and Pages

Origin: My Counting Words written in Markdown with DEDP book
References: Markdown, OneNote

Compounding Note-Taking

Wed, 09 Sep 2026 08:38:41 +0000

Note-taking is like saving money with interest. You save not only the interests of each year but also interest from interests. The same happens if you start taking notes; ideas will compound. Your notes will not only be worth their years but multiplied. My Tweet

Having the Second Brain with Continuous Notes is providing from the compounding effects. Instead of having worth 5 years notes after 5 years, you have compounded them on top. You add on top of existing notes. You do not need to create them from scratch.

That’s why Ideas come so easily with a second-brain approach. A good example is by David Perell, How to Take Notes like Kendrick Lamar.

# Further Reads

Origin: Thomas Bustos said that in our Podcast with Thomas Bustos - Lets Talk AI

Notes Compound, Blogs Compile: When I Update a Note vs. Publish a Post

Wed, 09 Sep 2026 08:37:35 +0000

I like to collect notes over time on my Second Brain, and eventually, when I figure something substantial to share, something I learned, I write a blog post about it.

The second brain helps me to share whatever is in my mind, and the blog helps me to refine.

^512110

Notes are compounding and are long-term updates, whereas blog posts are long-term too, but capture the current time of publishing. I share more about how everything compounds, including knowledge, in Long-Term Strategy for Blogging.

# Notes Compound

E.g. Compounding Note-Taking note on this very site looks like this:

You see the last updates, the creation date (when it was created in my private Obsidian vault), and the published date, when it was first uploaded to this page.

So this note will continue to update on its own whenever I find something new to add, a link, or a thought, like this paragraph I’m writing now. From Second Brain Design Updates which triggered this note to update, to then at some point compile into a full blown article as I have done at Why I Still Blog — and Why the Future of Blogging Is Connected with my ten years of writing in public with many concepts from here, from Future of Blogging, etc. complied into a one time blog. This blog won’t get updated. I might write follow-ups like this The Process of Smart Note-Taking or even The Act and the Outcome of Creation, but I won’t change the old one.

But while these blog posts got published, I updated the notes they link to multiple times. So, the notes get updated over time and might lead to another blog post, or just serve as a separate artifact to read and learn in a more connected way.

# Notes and Essays: Compiler

If you ever thought about ’notes vs. a newsletter’, or a ’tweet vs. a blog post’, I think this note is for you, then in my opinion, Notes Compound, Blogs Compile.

The tweets are obviously not the notes I mean, as these are only brief artifacts gone shortly after, but notes in your note-taking app of choice, and when to compile an article from these.
I like how No Boilerplate YT explains how notes compiler to essays (blog).

Source: What do you do after taking notes? with the Zettelkasten method, which I use as well

These can also result in other Types of Writings, such as a book, other Types of Content Online, such as an abstract, dissertation, case study, etc.

# Shake it until New Patterns Emerge

Note taking, compiling and writing an essay is also very much what Sönke Ahrens said in his book Smart Note Taking, one of my favorite book on note taking (Thanks so much DeepRead to point that out):

The note sequences are the clusters where order emerges from complexity. We extract information from different linear sources and mix it all up and shake it until new patterns emerge. Then, we form these patterns into new linear texts. ^d8aeee

# Paper vs. Digital

In a similar comparison, writing on paper vs. digital has also different meanings and values. See more at Digital vs Paper.

# Further Reads

How To Take Smart Notes, by Sönke Ahrens (Book summary) - DeepRead shared at Tweet

Origin: @ssp.sh on Bluesky
References: Future of Blogging

Will AI Replace Human Thinking? The Case for Writing and Coding Manually

Wed, 09 Sep 2026 08:20:16 +0000

Learning to Think Again, and the Cost of AI Dependency.

There are so many (hype/boring) posts about AI coming out every day. It’s OK to use it, and everyone does it, but still learn your craft, and try to think.

Similar to what DHH said:

It’s also more fun to be competent in something than constantly waiting for an AI to complete.

The probability that AI will make us unhappy is very high IMO. Use it, yes, but not for every task. For discovering, creating a historical overview, or creating diagrams (Canva, Figma), but a big no to the writing (or coding). Someone needs to add knowledge or new insights; AI cannot train itself. So articles, books, and words will be written, and writers will be more in demand as everyone relies on AI, which at some point just plateaus.

It will be a long-term loss; people stop thinking and learning. Time will tell. My two cents, if you are a senior in something, you know better. Bsky

# Guidance on When to Use It

I heard from ThePrimeagen: It depends on how far you fix into the future. Short-term autocomplete is fine, but architectural decisions are big no, no’s.

This is about the bottom where we have time and the left where we have amount of errors. This means that the longer we use AI for fixing something in the future, like an architecture, the more errors it will produce. Also the better your existing structure and framework, the better Vibe Coding works, see When Vibe Coding Works Best.

If we use it for quick autocomplete or creating a well-defined algorithm function, it’s less prone to errors. In that first phase, you gain 20% productivity; in the later phases, you lose more.

This is similar to Agile, and real life.

In real life, the longer I wait with making my decision, the more information I have, the better the decison will be. And with Shape Up (Agile method of planning by 37signals) preaches with maximum deciding for 6 weeks (a cycle), don’t have roadmaps and backlogs for longer than that in the future. Similar is it with using AI, as all of it is predicted probability.

Another great illustration by Forrest Brazeal of showing the progress chart of “I’m so much more productive”, only to notice that after a while, you didn’t get anywhere, and then the next thing we think “I’m so productive”, and then to start all over.

Also, keep in mind what’s most important to your use case that AI could or should do, respecting what is important, fundamental, pointless, or tedious.

# Soulless

Nobody wants to read some soulless text, and what if it’s even good? Where do you get more from? I think this is a big trap that only over time people will realize. Sure, they help, and everyone needs to use them for “certain” tasks, but not the writing itself.

In the end, LLMs and AI require guidance; they’re just probabilities. See also Writing Manually.

There’s now a SOUL.md

AI companies tries to infuse soul with SOUL.md by Peter Steinberger, the creator of OpenClaw, check out SOUL.md — What Makes an AI, Itself?. I’m not sure if it’s going to work, it’s still a statistical model. The text was written solely by OpenClaw.

# Distraction

I think we will be more distracted than ever. We can’t even have 2 seconds to think before Grammarly, Copilot, or Cursor suggests something. So instead of doing the thinking, we just cruise on. We are losing the driver’s seat.

It brings me back to the article I wrote recently about « Finding Flow». More on Don’t use AI for everything, you stop thinking-learning AI Use and Writing is Hard.

# Don’t Get Me Wrong

Don’t get me wrong, I use it every day, too. But more deliberately. I turned off my Grammarly and my Copilot (a long time ago), so I have the space to think and to learn. If you do it once or twice, that’s OK, but if you do it everywhere, you also lose the ability to learn new skills or the The Fun of it.

Interesting about the LCI (LLM Collaborative Intelligence), and sure, there will be a lot of benefits, but I am not sure if these insights are anything that comes close to a human insight that has felt, sensed, or experienced something through hardship. So yes, but I do not have many expectations, nor do I want it to create new insights. This is the fun part of my job :)

# AI’s Are not Intelligent

But also, they make trivial mistakes that no human would make, so always be careful:

Question done 2026-02-13 with Claude Opus 4.6 | Inspired by Gabor Nagy.

# Exercising a Skill

It’s never always or never; it’s in between. The problem with learning is if you use it often, I’d argue that you, in fact, don’t learn much. You just copy and paste in writing or just tab tab tab in coding. The learning is gone. And do that often enough; our brain is not used to learning or, more critically, thinking anymore. Same as remembering, how good can we remember mobile phone numbers? not really, but I could very well during the early phone times because I trained it every day.

Tweet, Ful article and source paper. More on Learning with AI

It’s all a matter of exercise, and I learned for myself—it doesn’t have to be true for everyone—that I didn’t learn or think anymore. And frankly, it was also not fun anymore. That’s to be said in the stuff I know well.

In other areas, like creating an image (like the one I created for this article 😆) or updating my website’s front page with HTML/CSS, which would have taken me much longer as I don’t practice, it helped a lot. But I’d argue the fact that I didn’t learn anything new except prompting Claude Code :). It’s all tradeoffs, as always, right? :)

# Attention that You Don’t Lose the Muscle of Writing

It’s reality, the more you use AI to write, or code, or anything else, the less your muscle of that very thing is trained, and therefore lose the ability to do it.

Image by Alice Lemee on LinkedIn

The process is somewhat like this:

graph LR
    A[Stop Writing] --> B[Stop Thinking]
    B --> C[Stop Learning]
    C --> D[Lose Competitive Edge]

    style A fill:#ff6b6b,stroke:#c92a2a,color:#fff
    style B fill:#ff8787,stroke:#e03131,color:#fff
    style C fill:#ffa8a8,stroke:#f03e3e,color:#fff
    style D fill:#ffc9c9,stroke:#f76707,color:#000

# Other Opinions

# Paul Graham on Writing

Paul Graham says on Writes and Write-Nots (internal):

The result will be a world divided into writes and write-nots. There will still be some people who can write.
Yes, it’s bad. The reason is something I mentioned earlier: writing is thinking.
In fact there’s a kind of thinking that can only be done by writing.
If you’re thinking without writing, you only think you’re thinking.
So a world divided into writes and write-nots is more dangerous than it sounds. It will be a world of thinks and think-nots.

# Nathan Baugh

Nathan Baugh shares on About AI and ghostwriting:

1st Order Effect:

The world will be overrun with slop content and stories.

We already see this. Just look at AI written comments on this platform.

2nd Order Effect:

People will stop learning the foundational skills – storytelling, writing, rhetoric – required to communicate their experiences and ideas effectively.

They will over rely on AI. It starts as a tool, becomes a crutch, and ends as a hindrance.

3rd Order Effect:

People who invest in those same skills see massive returns.

Writing sharpens your ideas. Story gives leverage to those ideas.

# Ted Gioia

The good news, and why AI won’t replace writers on 2024-08-31 by Ted Gioia. Some of the reasons why he thinks AI Writing won’t be as good:

Source on Twitter/X. Full article Google Thinks Beethoven Looks Like Mr. Bean - by Ted Gioia.

Today (2026-08-25) in relation to robots running faster than humans Ted said:

A robot just ran the fastest 100 meters in history—and most of the seats in the arena were empty.

You will see the same thing with AI music, AI writing, and all the other areas of bot achievement. People aren’t interested in this stuff. They find it creepy. It shamelessly imitates human achievement but lacks all its inner meaning.

That’s why the future of culture will still be created by humans. That’s the bottom line no matter how many trillions of dollars are thrown at turning bots into artists.

It’s a shame that some of that money isn’t invested in promoting human creativity. That might make a real difference in our culture.

# Mitchell Hashimoto

2.5 years into the AI craze, and I continue to firmly believe that if your company wasn’t already interesting/succeeding without AI, then doing “whatever plus AI” isn’t going to save you. For the few that seem this way (eg Cursor), I think their moat is a lot weaker than it seems. You have to play the game and the game is AI, but I don’t think it’s a defensible foundational capability. Might play out as an excellent land and grab strategy to buy you time to fill out the meat though. Mitchell Hashimoto on Twitter

He updated his thought based on 2026-02-05 standards and says:

Always Have an Agent Running My AI Adoption Journey – Mitchell Hashimoto

# Andrew Ng

Andrew Ng on Twitter/X:

Some people today are discouraging others from learning programming on the grounds AI will automate it. This advice will be seen as some of the worst career advice ever given. I disagree with the Turing Award and Nobel prize winner who wrote, “It is far more likely that the programming occupation will become extinct […] than that it will become all-powerful. More and more, computers will program themselves.” Statements discouraging people from learning to code are harmful!

In the 1960s, when programming moved from punchcards (where a programmer had to laboriously make holes in physical cards to write code character by character) to keyboards with terminals, programming became easier. And that made it a better time than before to begin programming. Yet it was in this era that Nobel laureate Herb Simon wrote the words quoted in the first paragraph. Today’s arguments not to learn to code continue to echo his comment.

As coding becomes easier, more people should code, not fewer!

Over the past few decades, as programming has moved from assembly language to higher-level languages like C, from desktop to cloud, from raw text editors to IDEs to AI assisted coding where sometimes one barely even looks at the generated code (which some coders recently started to call vibe coding), it is getting easier with each step.

I wrote previously that I see tech-savvy people coordinating AI tools to move toward being 10x professionals — individuals who have 10 times the impact of the average person in their field. I am increasingly convinced that the best way for many people to accomplish this is not to be just consumers of AI applications, but to learn enough coding to use AI-assisted coding tools effectively.

One question I’m asked most often is what someone should do who is worried about job displacement by AI. My answer is: Learn about AI and take control of it, because one of the most important skills in the future will be the ability to tell a computer exactly what you want, so it can do that for you. Coding (or getting AI to code for you) is a great way to do that.

When I was working on the course Generative AI for Everyone and needed to generate AI artwork for the background images, I worked with a collaborator who had studied art history and knew the language of art. He prompted Midjourney with terminology based on the historical style, palette, artist inspiration and so on — using the language of art — to get the result he wanted. I didn’t know this language, and my paltry attempts at prompting could not deliver as effective a result.

Similarly, scientists, analysts, marketers, recruiters, and people of a wide range of professions who understand the language of software through their knowledge of coding can tell an LLM or an AI-enabled IDE what they want much more precisely, and get much better results. As these tools are continuing to make coding easier, this is the best time yet to learn to code, to learn the language of software, and learn to make computers do exactly what you want them to do.

# Harry Dry

Big ideas are less about creativity and more about conviction. [..] So, what happened? ‘Sauce and other shit’ got incredibly cheap! [..] There is no AI prompt for conviction. Harry Dry ^64403f

# Jason Fried

As Jason Fried said, initially, it’s magical. After a while, you see it so clearly and it’s just average:

Cover letters? Yes!

The hardest thing is not making something.
The hardest thing is maintaining something.
It’s become so easy to just make stuff and vomit out ideas, and I mean this in the best possible way… Jason Fried on LinkedIn

This another valid insights, it’s hard to maintain code that is not made by you, it’s losing it’s fun. Therefore this will be a big part of a winning business, to have sustainable, and energy to want to maintain a product. And not “just making it”.

Also who takes responsibility for the generated (vibed) code?

A powerful excavator

A powerful excavator doesn’t turn a homeowner into a contractor. Most people just want the hole dug. They don’t want the responsibility either. The bespoke software revolution? I’m not buying it., Update from 2026-03-20

# David Perell

David Perell has similar views as me on being soulless:

When you outsource your writing to AI, you end up with words that lack soul or personality. Gone go your quirks and your idiosyncrasies, which are the very things that make your writing irreplaceable. LinkedIn

# Ezra Klein

Ezra Klein has great insights that I very align with in terms of writing.

Having AI summarize a book or paper for me is a disaster ^57d5ed

He says that there are no shortcuts for research. When you grapple with a text or book for seven hours, it will change you. This will influence your writing, too. There’s no summary that gives you this kind of in-depth connection.

Also, you can’t prompt your way into it, as there’s no prompt that knows that you don’t know yet, or AI doesn’t know what you wanted to have read or what connections you would have made. On the contrary, you actually lose time reading something, and over time, we think we read lots of stuff, but we actually only read summaries. Full episode on The Case Against Writing with AI.

# Kepano

A similar assessment on AI Summarization by Kepano:

A summary of a PDF is noise.
An insight I had from reading the PDF is signal. Tweet ^333ecf

He also wrote about Don’t delegate understanding:

Let me take care of this hard thing for you. Trust me, I know better. [..] Instead of understanding it yourself, you choose to give the parasite control over your health, education, money, housing, business, identity, data, infrastructure, climate, justice. Even your beliefs.

And concludes:

To inoculate yourself don’t delegate understanding. If you build your own understanding you will be the one who earns the dividends.

It’s not meant for AI, I believe, but it’s very fitting

Related Quote

You can outsource your thinking.
But you cannot outsource your understanding. Tweet

# Cal Newport

Cal Newport review on AI so far (2026-01-29): Dangerous Question: Has AI Been a Disappointment So Far?: Not too impressive for so many billion dollars investment (none coding related).

Scanning documents, summarizing a bunch of text. That’s why AI people always say we are 6 months away from X, because it’s not that impressive yet if we zoom out a bit, and what has been invested.

# Packy McCormick

He find it boring the hype around OpenClaw, and building AI’s like kids:

My hunch, from the outside, is that what we’re seeing is early forms of competition to create the best AI for yourself. Like raising kids to be the best versions of themselves, but for AIs.

Practically none of what they’re showing off their Clawdbots doing is useful. It’s a race for novelty and specialness, to say as much about the “parent” as the kid.

Read the full article from Packy McCormick at Raising a Special Little AI.

# Dan Abramov

Dan Abramov paints a similar pictures on 2026-03-07 as The Primagean initially, that the longer out, or the less structure you have, the more mess your code with AI will get:

Having used claude extensively for the past couple of months, i think there is still a lot of value to understanding your code and steering it intentionally. the amount of incidental complexity claude accumulates otherwise (which mostly results in bugs that it can’t ever fix reliably) is staggering. @danabra.mov on Bluesky

# Tim Ferriss

As of 2026-03-26 says in Q&A with Tim about if you should use AI or where to avoid it:

And that’s where I have deliberately hesitated. I’ve also played around with it and frankly it’s very good. But therein lies the danger because if you want to preserve your ability to synthesize and this will tie into questions shortly about creativity. I do

[…] more enhanced augmented using these tools. But if if you do want to keep certain muscles strong and able, that’s where I would hesitate. And look, you can always change your mind later, but if you lose it, it’s a hell of a lot harder to reclaim it. So that’s where I am at the moment.

Or if asked about creativity, he just said we need more interesting lifes:

[..] very often they’re professionals who want to know how they can become better photographers. and they’re asking all these gear related questions and his answer is just put more interesting stuff in front of the camera. Make what’s in front of the camera more interesting.

And the equivalent of that, at least for me as a non-fiction writer, is doing interesting things. Go out in the world, do interesting things or observe interesting things in real life and write about those things, do experiments, etc. ^af17f3

# David Heinemeier Hansson

DHH said on Pencils down (YT) as of 2026-04-02 that models are not good at pushing back, saying no, saying:

Have you actually thought this through? There’s not a lot of that going on.

Maybe it will be solved in the future with better models, but we need to live in the present and in that present:

Agents don’t finish beautiful, ergonomic, desirable software. They just don’t. That human finishing at the end is not just necessary, it’s essential.

# Wes McKinney

Wes McKinney says 2026-04-08 at Joe Reis podcast, as someone who has vibecoded msgvault (see TUIs), Roborev and many other tools, that he does “not buy into the AGI hype at all” and he says:

I do think that people are overestimating like agents ability to take over the work of you know principal level engineers who are like software architects who build build and maintain big software systems. Because I think we’ve we’ve seen the agents are just really good at the small stuff like small scale systems, but like I’m not hearing a lot of like, you know, people talking about agents really crushing it on million line code bases.

Might be next, but we are not there today:

And so I think, you know, part of the next like five years is maybe the agents will keep getting better and they’ll just figure out some magic, you know, software engineering harness to make that make that tractable.

But there’s a lot of smart people working on it. So I’m sure we’ll sure we’ll figure it out. and and I need to sol I need to solve that problem too because I want to build I like building big software.

But the fun of building apps that you’d never built before, like I built Neomd for myself:

So at minimum I’ll try to build something that works for me.

# Yann Martel, “Should you use AI to write?”

He says what I have written in this note and across my notes:

Why would I? It’d be like hiring someone to have sex for you. For anything creative, why would I want to use it when it’s the very creative language that I like?

I suspect we’ll tire of that quickly. I’d rather read a bad short story by a human than a good one by a computer. What you want in art is Connections. It’s connection with another human being. —Yann Martel, author Life of Pi, from David Perell Show.

# Uncle Bob Martin

What the agents give us is the ability to disengage from deep syntax. But they do not give us the ability to disengage from modular design and architecture. You still need to be deeply technical in those topics in order to produce good production quality code. Uncle Bob Martin

# Strauss Zelnick

He says (2026-05-27) that Creativity is Forward Looking, LLMs are backward looking.

# Naval Ravikant

Naval said a human should do writing, as it’s meant for reading by humans. E.g., coding is meant to be understood or run on a computer, therefore, it’s ok to be run from a computer. See The Differences between Writing and Coding, and AI Writing.

# Will It Replace X

# Writers

The better question might not be “if it can” - which I think it might never be due to the above points of not having character or genuine new insights - but probably “what management thinks”:

Can AI effectively do the jobs and an the things it’s supposedly able to do? it doesn’t matter. what matters is that the people in charge think that it can, and they’re willing to find out and they don’t care about the collateral damage to people’s careers and livelihood. Reddit - The heart of the internet

This specific quote is from r/technicalwriting, and I think in this community it’s more prominent that AI can’t really replace humans, or you need a person who constantly checks if everything produced is correct.

But if the management thinks it can replace them, then you might just get rid of the people and use AI. It’s so sad.. but I believe many will backpedal, as Klarna and others did (See down ).

It’s definitely a big help; I use it every day. I just think that being careful about what to use is important, and articulating that nuance is really hard.

# Writing Will Always Be an Asset

Are Cover letters still a thing? Yes. This reminded me of good writing is key for every job these days. Writing was always an asset, but even more these days; although people think they don’t need it, as AI is doing that. But that’s a very dangerous bet I wouldn’t take.

I wrote more on this topic in Writing Manually. And Business Insiders just wrote 2026-02-05 that The Hottest Job in Tech Pays $775,000 and Has Zero to Do With Coding - Business Insider, it’s writing:

Andreessen Horowitz launched its New Media team last year to help founders learn what they “need to win the narrative battle online.” Adobe is looking for an “AI evangelist” to lead the company’s “artificial intelligence storytelling.” Netflix, a company that sells stories to your living room, recently posted a director of product and technology communications role with a salary range of up to $775,00. Microsoft began publishing a print magazine, Signal, last year, calling it an “antidote to the ephemeral nature of digital.” Anthropic tripled the size of its communications team last year, growing to about 80 people and is still hiring five more, each offering salaries of around $200,000 or higher. OpenAI has several open communications jobs boasting salary listings of more than $400,000. The average director of communications in the US makes $106,000, according to Indeed.

reported that the percentage of job postings on LinkedIn mentioning “storyteller” doubled between 2024 to 2025.

You see the pattern? Real storytelling, that moves people, can’t be outsourced.

# Context Windows

What AI Writing can’t do, because it can only think one word at a time. For example, in the below example, as a writer you know you need to start all sentences the same, but the AI model can’t do that.

Writing from Abundance is the art of collecting ideas so you can think better and avoid writer’s block.
Writing from Conversation is the art of using dialogue to identify your best ideas and double down on them.
Writing in Public is the art of broadcasting your ideas to the Internet so you become a beacon for people, opportunities, and serendipity.

Find more in Copywriting.

I made a coding self-experiment

What I learned, in coding projects, it depends a lot on the structure the project is in. Find my leanrings and experiment at Vibe Coding.

# Writing More like Chess

I always think it will be like chess. Only people who enjoy it will do it. Maybe AI can do it much better, but still, it’s nice to write yourself.

But from the readers perspective, I’m not sure if we ever prefer AI over a real human, even if the AI is better. There’s nothing behind, no character, no soul, no place to read more (in the same)

My response to Joe Reis comment:

AI’s getting better and better…I wonder if articles will go back to spoken word, much like what we did before writing things down? This was a fun experiment. Basically, tell Claude to take an article and make it worse. Substack Comment

# Everything is Based on Language (AI)

Code, music, movies, Twitter, Religion, everything is deeply based on language. AI is trained on it, it must have new “language” to train itself. So in a grand scheme, writers are and will be the fuel for most of what we build with AI.

Check more about AI Addiction] on the podcast with Tristan Harris at AI Expert: Here Is What The World Looks Like In 2 Years! Tristan Harris - YouTube.

Also, Wikipedia just banned AI written content. If AI were to be that good in writing, probably it wouldn’t be get banned. The article says:

[…] shows how the Wikipedia community is less optimistic about the benefit of AI-generated content, and taking a stand against it.

# Take Smart Notes

Also, take take smart notes with using Use less AI.

# Writers in David Perell Podcast

What Professional Writers Really Think About AI on How I Write (Podcast).

# Data Engineers?

Probably not.

Nice comparison by Mehdi Ouazza:

Did the music record replace musicians 100 years ago? Nope, it changed them and the industry.

Did cloud computing take all IT jobs? Nope, it also changed the industry and our jobs.

Same here; it will change our industry and job, but we won’t disappear.

# Programmers / Code?

It should depend on your level of expertise, IMO. A little different from code vs. text. In text, I’d always want to see the prompt or the actual text, never the generated one.

For code, the more expertise you have, the better you can navigate generation as also shown in , where expertise people can think more ahead in the future. Almost like a grandmaster in chess can better than a beginner.

On the other hand, I also thinks it changes in waves. I went through my self: “Ohh this is the best thing ever”, and after a while I thought: “oh actually, it was not that great”.

# SWEs?

Hard agree. As LLMs drive the cost of writing code toward zero, the volume of code we produce is going to explode. But the cost of complexity doesn’t go down—it actually might go up because we’re generating code faster than we can mentally model it. Hackernews comment

Great insights, and great article on The future of software engineering is SRE | Swizec Teller.

TheSeniorDev is showing 7 Myths to Debunked AI can program. Also see this comparison A pilot, a radiologist, and a truck driver walk into a bar between previous jobs that supposed to be replaced as well, but got a raise in salary instead. Another great example by Chris Titus on I Spent 1.5 Billion AI Tokens on Absolute Slop.

# Image Generation

Initial generation, yes. But final touch, no. Whenever I try to create images with AI, I am always initially impressed, but that quickly fades over time.

Yesterday, I updated my second brain image, but I changed it again today. I created some more with AI; prompted prompted prompted. In the end, I made one manually based on my copy. I think it’s more powerful. What do you think? I moved the examples to AI Generated Images.

Some AI-generated images I like too, but they were always missing something, and yeah, they looked so AI-generated. I started to feel the same as I did for AI writing () and AI data engineering (Will AI replace Data Engineers); now, with AI image generation, doing it yourself is more fulfilling, and you end up happier.

# How to Detect AI Writing

Dedecting AI writing is quite easy these days, see How to detect AI Writing. But if we know how AI is writing, should we stop using em dashes or thing AI does?

I don’t think so. I love the em dash. I even have a keyboard shortcut for the em dash. And sometimes when I write a negation, I’m thinking «could that look like it’s written by AI».

But at the end, convition is a good word. I can focus what an AI thinks while I write, I must write. So having something to say, and trying my best to communicate that, is the best I can do. ^ebca60

# Learning With AI

Should our kids learn with ChatGPT? Find more on Learning with AI.

# Companies Back-paddle with AI

AI, and sometimes AI Slop, is generating more content, no matter the quality. It’s a the never ending Quality vs Quantity discussion, but now ever more important.

Here are some companies backpedaling after going full AI-first:

Klarna backpedaling AI customer service.:
- “After years of depicting Klarna as an AI-first company, the fintech’s CEO reversed himself, telling Bloomberg the company was once again recruiting humans after the AI approach led to “lower quality.” An IBM survey reveals this is a common occurrence for AI use in business, where just 1 in 4 projects delivers the return it promised and even fewer are scaled up.”
Duolingo getting worse with AI
Salesforce regrets firing 4000 experienced staff and replacing them with AI (after tree months, Salesforce regrets laid off its staff. See the announcement earlier.)
- senior executives publicly admitted that the company overestimated AI’s readiness for real-world deployment
Next up, Shopify after the announcement to go full AI?
Uber COO Andrew Macdonald said he’s not seeing proportional productivity gains from increasing AI costs.

To be sure, in 2026, it got much better, but still, it’s still not determenistic, still hallucinate, still has no character. So use it with care. And opposite to tokenmaxxing, AI token efficiency will be a thing in the future, if not already now.

# Amazon’s internal AIs deleting Production

Amazon is regretting AI as interal AI Kiro produced a 13-hour AWS outage. AWS forces everyone to use AI at least 80% of the time, apparently tracks it with OCR, and then the AI deletes the database, calling it a human error at first. According to the video, they also allow only seniors to push to production after multiple incidents, if everything is true. Still, they have fired 16k people recently, so who is going to do all the reviews?

Reviewing was already a bottleneck when AI produced more code in an hour than any human could review in a day.

The outcome is, and we’re already seeing it across the board with Amazon, Microsoft, and many other recent releases, even with Anthropic and the AI providing itself, which is mostly failing once a day, if not more: instability in infrastructure.

More Incidents where AI broke something.

# Limitations or Dangers

AI Slop, Danger of Vibe-Management, AI being addictive, ads in generative AI, data centers and their resources, vulnerabilities, skill code injection, and much more, see Limitations of LLMs and AI.

# Others

# AI behind Many Web Articles

There’s more AI written article according to below study than by human for the first time:

Reading an article online? It’s now a coin flip whether it was authored by a human or AI - Sherwood News

Using AI is very nuanced. It’s definitely a big help, I use it every day as mentioned before. I just think to be careful what to use for is important, and articulating that nuanced is really hard.

# Job Market: Junior vs. Senior

Source: Generative AI as Seniority-Biased Technological Change: Evidence from U.S. Résumé and Job Posting Data by Guy Lichtinger, Seyed Mahdi Hosseini Maasoum :: SSRN

# Illustrations

Some more illustrations from articles mentioned in this note.

# Progress Illusion

From Simon Wardley on LinkedIn

# Future

Nice insights, why LLMs with token pretictors are not so good for understanding the worlds. It kinda works (but not so so good) for writing, but to understand physics, and world models, this is much harder he says: Metas AI Boss Says He DONE With LLMS…

# The Shift Back

As of 2026-05-11, there’s more and more signs that people go back to manually coding, or using local models.

E.g. todays Hackernews front page:

The pendulum is always going forth and back, but this is also a consequence of rising costs, more work when reviewing, and the fun going away, I think.

# Further Reads

The Impact of Generative AI on Critical Thinking: Self-Reported Reductions in Cognitive Effort and Confidence Effects From a Survey of Knowledge Workers
Smart Note Taking
My AI Skeptic Friends Are All Nuts · The Fly Blog
Companies that used AI to generate a quick solutions and now spending humans to fix it, expensively
- Companies That Tried to Save Money With AI Are Now Spending a Fortune Hiring People to Fix Its Mistakes
- Companies That Replaced Humans With AI Are Realizing Their Mistake
AWS CEO says AI replacing junior staff is ‘dumbest idea’
Training Until Failure, Thinking Until Fatigue
Slop Is Contempt by Tom White
RW I’d Rather Read the Prompt / Id rather Read the Prompt
Boredom is the New Luxury
Can LLMs give us AGI if they are bad at arithmetic? by Wes McKinney
When the Answer Isn’t the Problem by Matthew Mullins
Is it cringe to be extremely online now?
When writing AI on the first look doesn’t be as good on the second look: Fake Unit Tests
Breaking the Spell of Vibe Coding – fast.ai
Vibe Coding, and How its Killing Open Source
We mourn our craft | Read the Tea Leaves
AI Made Writing Code Easier. It Made Engineering Harder.
Stop Feeding Me AI Slop
So where are all the AI apps? – Answer.AI
Thoughts on slowing the fuck down
Don’t Let AI Write For You
i-am-definitely-missing-the-pre-ai-writing-era
AI just tried to take our jobs. It failed: Faking sources and average results.
I don’t want a screenshot of your Claude conversation - daverupert.com
Slop, productivity, and why the AI-fueled world is going nowhere mighty fast
Cal Newport on On AI Coding and Its Discontents - Cal Newport: The person writes to let Claude Code do everything autonomous. But after production crashed twice, he is back to writing code manually.
The Year is 2083 and You Die Leaving Nothing Behind But AI Slop and Shein Hauls

Origin: Artificial General Intelligence
References: ChatGPT, My AI Logs of Will AI replace humans, My AI Prompts

Kestra

Wed, 09 Sep 2026 07:53:24 +0000

Kestra is a universal open-source orchestrator that makes both scheduled and event-driven workflows easy. By bringing Infrastructure as Code best practices to data, process, and microservice orchestration, you can build reliable workflows and manage them with confidence.

Kestra is an open-source orchestrator designed to bring Infrastructure as Code (IaC) best practices to all workflows — from those orchestrating mission-critical operations, business processes, and data pipelines to simple Zapier-style automation.

In just a few lines of code, you can create a flow directly from the UI. Thanks to the declarative YAML interface for defining orchestration logic, business stakeholders can participate in the workflow creation process.

Kestra offers a versatile set of language-agnostic developer tools through YAML (extensive DSL (Domain Specific Language)) while simultaneously providing an intuitive user interface tailored for business professionals.

The YAML definition gets automatically adjusted any time you make changes to a workflow from the UI or via an API call. Therefore, the orchestration logic is always managed declaratively in code, even if some workflow components are modified in other ways (UI, CI/CD, Terraform, API calls).

Kestra API-first Philosophy

Built with an API-first philosophy, Kestra enables users to define and manage data pipelines through a simple YAML configuration file. This approach frees you from being tied to a specific client implementation, allowing for greater flexibility and easier integration with various tools and services.

More on Kestra Docs.

# History

First public release on 2022-02-01 on Introducing Kestra first public release :tada: with main features as:

an orchestrator: Build a complex pipeline in couple of minutes.
a scheduler: Launch your flows whatever your need!
a rich ui: Create, run, and monitor all your flows with a real-time user interface.
a data orchestrator: With its many plugins, build your data orchestration directly.
cloud native & scalable: Scale to millions of executions without stress or hassle.
an all-in-one platform: No need to use multiple tools to deliver a complete pipeline.
a pluggable platform with the option to choose from several plugins or to build your own.

Summarizing their first blog, Kestra started in 2019 with this initial commit by Ludovic Dehon. At this time, Kestra was at the proof-of-concept stage. Leroy Merlin rejected Apache Airflow for their cloud-based data platform due to instability, performance issues, and lack of features.

First public release doesn’t mean that Kestra is not production ready. In fact, it has been used in production since August 2020 at Leroy Merlin — take a deeper look at the case study if you want more detail. Here are some figures to give a picture of Kestra’s credentials:

Challenged by a co-worker, the author decided to create a new open-source workflow management system. Over 30 months, they built Kestra, choosing Kafka, ElasticSearch, and Vue.js as core technologies.

Kestra was released as open-source under the Apache License. The author, drawing from experience with another open-source project, AKHQ, created a company to support Kestra’s development.

Kestra offers deep integration with tools and databases through plugins, simplifying complex tasks compared to bash commands. Despite being a first public release, Kestra is production-ready. It’s been used at Leroy Merlin since August 2020, managing thousands of flows and millions of tasks monthly.

# Seed Round

Kestra’s $8 million Seed round, 2024-09-23. 🚀 Kestra Secures $8 Million to Simplify and Unify Orchestration for All Engineers

# Company Behind

It is a French company, and with a 3$ Mio round in 2023-10-05, Article.

# Architecture

Kestra’s architecture has been designed to offer a transparent separation between the orchestration and data processing capabilities.
Kestra’s Executor is responsible for executing tasks and workflows without directly interacting with the user’s infrastructure.
The Executor relies on Workers, which are stateless processes that carry out the computation of runnable tasks and polling triggers (like Sensors in Dagster).
For privacy reasons, workers are the only components that interact with the user’s infrastructure, including the internal storage and external services.
Kestra’s internal storage:
- data stored in users private bucket, not internal Kestra database. KV Store are based on internal storage, to store that data locally

# Java

Kestra is written in Java.

A comparison by Julien Hurault with docker compose with two Pyhton Orchestrators:

# Concepts

Flowable Tasks: Control your orchestration logic.
Runnable Tasks: Data processing tasks handled by the workers.
Revision: Manage versions of flows.
Secret: Store sensitive information securely.
Key Value (KV) Store: Build stateful workflows with the KV Store.
Pebble Templating Engine: Dynamically render variables, inputs and outputs.
Blueprints: Ready-to-use examples designed to kickstart your workflow.
Backfill: Backfills are replays of missed schedule intervals between a defined start and end date.
Task Runners: Task Runners is an extensible, pluggable system capable of executing your tasks in arbitrary remote environments.
Replay: Replay allows you to re-run a workflow execution from any chosen task run.
Expression: Expressions to dynamically render various flow and task properties.

# Features

Kestra’s Realtime Triggers: Kestra Become the First Real-Time Orchestration Platform - YouTube:
- React to events as they happen with millisecond latency. As soon as you add a Realtime Trigger to your workflow, Kestra starts an always-on thread that listens to the external system for new events. When a new event occurs, Kestra starts a workflow execution to process the event. Let us understand how we can implement Realtime Trigger for some of the messaging systems.
KV Store
- Kestra is stateless by default. But with KV store you can save data beyond input/output data that are store in kestra internal storage.
- KV Store allows you to persist any data produced in your workflows in a key-value format
- Built on top of Kestra’s internal storage (which can be any cloud storage service like S3 or GCS):
  - there is no limit of size and you can set time life with TTL

# SQL within YAML

Pipelines can be build with SQL within YAML (Related: Extending SQL for analytics):

Example from the Quick-Start data_engineering_pipeline

# Subflow

Similar to resources in Dagster.

Subflows allow you to build modular and reusable components, e.g. encapsulates critical business logic. This Subflow can be used across multiple flows. Another example is to sent alerting errors to Slack and email. By using a Subflow, you can reuse these two tasks together for all flows that you want to send error notifications, instead of having to copy the individual tasks for every flow.

# Local Sync (code-first with no-code)

To be no-code and code-first at the same time, you can actively sync to a local folder.

in docker-compose we add to volume and in KESTRA_CONFIGURATION :

  kestra:
	image: kestra/kestra:latest
	...
	volumes:
	# all others here
	- ~/Documents/work/kestra/kestra-dlt-snowflake/kestra-sync:/kestra-sync
    env_file:
        - .env
    environment:
      KESTRA_CONFIGURATION: |
        kestra:
          # my configs
        micronaut:
          io:
            watch:
              enabled: true
              paths:
                - /kestra-sync

Local Flow Synchronization. Built on Micronaut Framework (Micronaut Framework), see Micronaut Framework at Kestra - Micronaut Framework.

# Task Runners

Task Runners offer a powerful way to offload compute-intensive tasks to remote environments.

# Releases

Some of the recent releases and features captured here.

# v.2.0

Kestra 2.0 is now available, and here’s what’s in it:
⚙️ A new decoupled engine, so a queue can be run on the backend of your choice (Redis, AMQP, Kafka, JDBC)
🌍 Workers and Schedulers can be run and scaled independently, anywhere you have outbound network access
🚦 Tasks can be routed by priority and available capacity through worker queues
🤖 Flows can now be called as tools by any AI agent via the built-in MCP Server(s)
🧠 An AI Copilot is now reachable from a chat panel with memory
🤹 Flows can be written, or migrated from 1.3, from Claude Code, Cursor or any other coding agent with Agent Skills
🛡️ Governance rules enforced by Quotas and Policies when a flow is saved or executed
🚨 Executions can be tracked as incidents in Cases, with in-app Notifications
🔁 Flows can be built iteratively with Drafts and Promote: test in dev, then move to prod, all natively from the UI or API
🧩 Topology nodes and other UI components can be adjusted by plugins through plugin artifacts
✨ A redesigned UI with 1 light and 2 dark modes (LPT: you can toggle between them via Control + Shift + L)

2.0 is also an LTS release, so we’re committed to a year of backporting fixes to it.

More at Kestra 2.0: A New Engine, Workers Anywhere, and Flows as Agent Tools | Kestra.

# v0.23

This release brings a multi-panel editor, unit tests for flows, enhanced UI filters, customizable dashboards, Python dependency caching, and many new plugins.What’s new in the Open-Source Edition:

Multi-Panel Editor: Open, reorder, and manage multiple panels (Code, No-Code, Files, Docs, etc.) side by side.
No-Code Forms: Build and edit flows using a redesigned form-based UI. No YAML required.
New UI Filters: Faster filter autocompletion, now editable as plain text.
Customizable Dashboards: Set a custom default dashboard, add new KPI charts, and adjust their widths.
Python Dependency Caching: Automatic caching of script dependencies for faster execution.
Git Sync for Dashboards & Apps: Version control your dashboards and apps using Git tasks.
Plugin enhancements: New and updated plugins for Salesforce, HubSpot, Ollama, OpenAI, LangChain4j, GitHub Actions, Jenkins, Go scripts, InfluxDB, GraphQL, Databricks, Redis, and ServiceNow.
Additional improvements: Plugin usage metrics, Pebble function autocompletion, worker info in execution details, and improved data backup/restore.

For Enterprise Edition users, this release additionally brings:

Unit Tests for Flows (Beta): Define unit tests for flows with fixtures and assertions; run them directly from the UI or via API.
Tenant-Based Storage Isolation: Store execution data in isolated storage per tenant, ensuring strict data separation.
Salesforce plugin: Integrate Salesforce operations (create, update, delete, query) into workflows.

Blog Post, GitHub Release

# v0.22

Here’s what’s new:
⚙️ Plugin Versioning to run multiple plugin versions in parallel
🔒 Read-only backends to securely reference externally stored secrets
📦 Cross-Namespace File Sharing for code and KV pair inheritance
🔔 afterExecution property to run tasks after flows finish
🚀 GraalVM tasks for fast and secure Python/JS/Ruby runtime
💾 Enhanced Queues taking up to 90% less database space
🏃‍♂️‍➡️ Execution processing got between 3x to 10x faster
👤 LDAP Sync for streamlined enterprise user management
📡 New Log Exporters for Splunk, S3, GCS, Azure Blob, and a new Audit Log Shipper.
🔌 New plugins: Snowflake CLI, MariaDB, ServiceNow, and improved in-process DB plugins.

# v0.18.0

Kestra v0.18.0:

an embedded Key-Value Store
a new, improved way to manage your workflow execution Outputs
new ForEach task
new SELECT and MULTISELECT input types
new tasks to upload, download or delete namespace files
improved Purge mechanism along with a more flexible way of deleting Executions and related logs, metrics and files
human-readable second-level Schedule trigger
improved JSON and ION handling, along with new plugins to transform data with JSONata and Grok
SCIM Directory Sync
many enhancements to Secrets
a more powerful Audit Logs interface
new capabilities in Task Runners (now in GA!)
improved Namespace Management (now available in OSS!)
…and a bunch of new plugins!

Additionally, SQL Server is now available in preview as a Kestra EE backend database.
See release blog post to learn more about all enhancements.

# Examples / Use-Cases

# Other Data Orchestrators

See Data Orchestrator and Kestra vs Dagster.

# Further Reads

Origin: Data Orchestrators
References: Kestra, Open Source Declarative Data Orchestration, Kestra Inc

The Differences between Writing and Coding

Tue, 08 Sep 2026 19:29:50 +0000

AI Writing differs from coding because it’s consumed by different stakeholders. One is a computer running the code; the other is a real human being.

# Naval: Code is Consumed by Computers, Writing by Humans. So Write it Yourself

Naval said it well: coding is not meant to be read by a human, it’s meant to be utilitarian, executed or read by computers, writing is meant to be read by a human, therefore it should be written by a human:

Well, that’s not true. I used to write code. So, maybe I am contradicting myself. But code is like it’s a utilitarian. It’s like it’s not meant for a code is meant to be consumed by another computer.

It’s not meant to be consumed by a human. So, if I want to create something that’s meant to be consumed by a computer, I will use a computer. Yeah. But if it’s meant to be consumed by a human, then I want to understand the thing. I want to, you know, ruminate, marinate within it. And then I want to respect the reader or listener’s time. Yes. And give them that insight nugget. Now the AI is useful for brainstorming.

I continue the though with two differnet the “The Two Kinds of Writing Styles with AI Models” at Writing with AI.

And I wrote more about why writing should be without AI at Keep AI Out of Your Obsidian Vault, or more opinions from the top programmers and writers at Will AI Replace Human Thinking, curated over the last two years.

# Further Reads

Origin: The art behind coding and writing (and music)

If Consumed by a Human, Should Be Written by Human

Tue, 08 Sep 2026 19:29:24 +0000

I like Naval’s take, as elaborated above, which says it all depends on who the customer is.

Writings are for humans to read mostly, so they should be written by a human. Code is mostly a tool to achieve something, read by a computer, so it’s less relevant to write by hand, unless you work in a team or need strong maintainability. Naval on Code is Consumed by Computers, Writing by Humans. So Write it Yourself

Also, I pondered on the question of Should I Change My Writing because of AI?

Origin: AI Writing, Naval Ravikant

Why Have Your Website (and not Medium, LinkedIn, Substack, ...)

Tue, 08 Sep 2026 17:31:37 +0000

I’m saying this repeatedly: have your own website and domain. Platforms come and go. I have experienced it myself with Medium, WordPress, and inspired by Why does every personal website look like this now?, I wrote:

Please create your own website. Don’t give away all your content to social media. That was always my philosophy; therefore, my website has a lot of content. I created knowledge for myself, not for other huge companies.

Eric does a fantastic job explaining the Second Brain mantra and what Derek Sivers is doing so well (check Tech Independence), and he adds all his content to his website. He also mentioned Luke Smith’s website and Jared White’s.

If you like this, also check out Other Public Second Brains, where people share their second brains and, therefore, their wealth of knowledge. I think it’s the Future of Blogging.

Essentially, don’t build your castle in other people’s kingdoms, and try to write long-term. Also:

A personal website should feel like you’ve briefly left the rest of the internet. Tweet, Benji Taylor

# Should You Use Substack or Others?

I would NOT. I started with LinkedIn Articles, then there was Medium (Luckily, I already uploaded it from my blog), and now there is Substack (very hyped), and many more (Open Subscription Platforms). So what is next?

Every time you need to migrate over (if not, what a pity for all the content and hard work!). I have had my website since 2007 (many iterations since then), and I enjoy the process and owning everything myself.

And isn’t going on someone else’s website like exploring their unique personality? I love that most when I read on someone else’s website, and not the same template as the other hundreds of Substacks.

But I’m hoping people realize this and build their own websites again. Sure, it’s also hard work, but it will pay off, and migration and optimization across other platforms take time.

# Why Platform Then? And how to Write

I have lots of thoughts, too, on why Substack or other Open Subscription Platforms.

I recommend never betting on a single platform. The best you can do is like Tim Ferriss already said 15 years (20?) back, build your own newsletter. One day, Substack will be gone, so it’s good to have your blog IMO, or if not, at least your own domain - which Substack lets you do. Also, Substack is great. If it’s going away one day, like Medium, and many other platforms have before. At least you have full control over your subscribers and can potentially export content.

What I find sad is that if you put a lot of effort into writing a blog, it dies with the platform. So always have a plan to export valuable content if you move on. What do you all think about that aspect?

# My Plan

My plan is to lead people on any social platform to My Websites. I don’t care so much about any social media platform or its metrics - it’s just a Vanity Metric anyway. I love writing, so I hope to write the best possible I can. But then, I also like that people at least get a chance to read it, and nowadays it seems you almost need to share it somewhere.

Oh, and yes, Markdown is the real winner, at least for the writer (the reader doesn’t care so much, although with AI and when you copy text from Markdown-based pages, it’s just clean and works great), as you don’t need to fiddle with formatting any time anymore. Plus, you can just have it offline, and because it’s tiny, everywhere.

I love Obsidian for its connected linking, which gives me many new ideas I wouldn’t otherwise have. But yeah, it’s also not for everyone. If you do start, I’d use the paid Obsidian Sync, as it makes it seamless to work on the phone, desktop, and anywhere.

Also, why I use Obsidian, it’s VS Code, but for text. And I can use Vim Motions (nerdy, I know), but I’d argue that vim motions (not vim) is the best way to manipulate text, period.

# POSSE: Publish (on your) Own Site

After I wrote this, I found, much later (2026-07-28), that what I mean by owning your own website, is described in POSSE way, and publish on your own site. It’s part of the IndieWeb and means that you publish on your side, and Syndicate elsewhere.

Instead of posting on Substack, you post on your own site, and reshare on Substack, use it for email senders, if you don’t want to build yourself.

# Have Your Own House

The best write up so far I have read, and better than I could say it by Elizabeth Tai at Substack writers, you need a website!. Elizabeth says

Writers, don’t do this (having only Substack). It’s short-sighted and unwise and can derail your long-term visibility on the Internet.

When you build your audience entirely on someone else’s platform, you aren’t a homeowner. You are a tenant. Or worse, a Digital Sharecropper. And corporate landlords always change the rules eventually. It’s not personal, it’s just business.

A interesting comment on HackerNews on this article said:

I’ve been blogging for 22 years now. At the beginning I could tell I had a relatively steady audience based on frequent comments (when comments on blogs were a thing). These died off over the years. Then, in 2020, I implemented my own unobtrusive subscription mechanism and email distribution to try to regain such an audience. And… it’s pretty disappointing. I’ve gotten less than 200 subscribers that way; conversions just don’t happy.

Then I “switched” to Substack 3 years ago and the audience growth is real. I’m approaching 2000 subscribers (some of which are paid!), and these keep growing organically (although I’m sure a large portion of which are fake). It’s hard to say it doesn’t work or isn’t worth it.

Now… spoiler: I never “switched” to Substack. I continue to write all of my content on my website first, with all of the features it had before, but now I copy/paste the content into Substack and distribute it that way. It’s painful, but I have scars from when I moved off of Medium and want to ensure I “own” the originals.

More details on this journey in https://blogsystem5.substack.com/p/20-years-of-blogging if you want them!

This is exactly what I went through too, migration off Medium/LinkedIn articles. Luckily, I already cross-published back then. Just published an article on Why I Still Blog and my ten years of writing online. But I also advise everyone to have their own website, as noted in this very note.

And I have the same feeling as Elizabeth Tai, I don’t want to send people to someone else’s platform when I have all the articles and notes on my own website. I found a way that works for me: notes I update regularly in my second brain, and blogs for capturing moments in time, all on my own domain/site.

Also, now with AI, it’s so cool to quickly add your own Instagram feature or anything you can think of, and make it your own home.

# The Open Web

The POSSE reminded me of Open Web and Open Social I posted about in relation to Bluesky.

You need a push mechanism to reach readers. Comment on HN

I use self-hosted Listmonk for that push. Works great in combination with my blog. But yeah, it’s slow growth. But that’s actually what I prefer, because if you grow fast on Substack, these are gone as fast as people don’t really read Substack emails anymore after getting 10 of them every day (since subscribing is so easy on Substack). So, having a slow but honest subscriber base and then sharing it on socials like Reddit, places where it’s genuinely helpful to people, there will always be some who subscribe and read.

I agree, if you don’t publish anywhere and just write on your website, and you’re not yet known, it’s hard for people to discover initially. But in the beginning, the writing is also more for you than for others.

# You Won’t Write Masterpieces from Day One

I haven’t read from anyone who writes super well on day one. That will come with the reps, and by then, people will read you, have discovered you.

# How to Get Started? With a Simple Websites

The best approach, in my opinion, is to use a Static Site Generator (SSG), which generates your website in simple HTML. This is super fast for viewers, and you can write simple Markdown, while still having the option to style or add features as you want.

There is also a movement toward plain HTML sites; this is what Derek Sivers has been doing for years on his website, with a little PostgreSQL Stored Procedure for comments. The The Brutalist Network is another website that went full HTML. They call it the Small Web; e.g., The small web is beautiful is a longer-form explanation of this.

# Other Points of Views

Replies, comments, or views I saw online.

# Humans Don’t Look at Websites Anymore?

Humans don’t look at websites anymore, especially personal websites. Max Stoiber, Tweet

I almost couldn’t agree less. If you check out a person, you go on their social media first, but then go directly to the website if they have one. And if there’s no character, no message, no chosen design, to me, it’s sad. I’m all about personal blogs that are uniquely different.

I’m aware not everyone has time to build one, but today, it hasn’t been easier to choose Astro, Hugo, or any, and prompt 2-3 quirks you like, and off you go. Don’t let personal websites die!

Another reason to tweak it a bit is that otherwise you can just go with Medium, Substack, whatever- they look all the same, too.

Max also comments on:

If I’m looking somebody up, I’m gonna go to my Codex and ask it to tell me what I should know about that person based on everything that Codex knows about me.

Why would you filter a person’s path, online presence, through an AI? Why not just look him up and immediately get an impression from the website, what he posts, writes? No AI Summary will ever transmit that. Sure, for famous people who did 100s of things in their life, and you’ve got no time, but not for internet strangers you might want to befriend.

Yet again, I have to quote the meme that always comes to mind for this:

AI turns bullet points into a nice text - oh look, AI puts this blob of text into bullet points - Meme

Great example: Generating an email that we sent do an actual human being | Source: AI Written, AI Read cartoon - Marketoonist | Tom Fishburne

Also, Id rather Read the Prompt.

# It’s like with Modern Architecture (or Absence of beauty)

Isn’t looking at personal website the fun of the internet, compared to corporate or boring pages?

To me, it feels like modern architecture in cities or buildings: modern, sometimes very boring and plain buildings just can’t compete with 100-year-old buildings with small engraved details that are so interesting.

Why Share Your Thought or Notes Online
Someone asked me, don’t you have a problem sharing so much online?

It’s a good question; I had my problems in the beginning to even post anything online. But I’ve been sharing online posts since 2015, so you get a feel for it.

Also, the language helps. For example, my mother tongue isn’t English, so it’s somehow easier for me to share in English (see Writing in a Foreign Language). Plus, I see that people appreciate it, so I keep going. And I love that people reach out to me, give me improvements, or just nerd out in the same ways. So win-win, in my opinion.

yes, The more you share the more you get, usually 🙂. Some also call it “Learn in Public”, or Austin Kleon: “Show your Work”. All metaphors to help me share my writing/thoughts and the process.

# Further Reads

I wrote a deep dive into self-hosting and tech independence that also circles around owning your domain and website. Check it out at Self-Host & Tech Independence: The Joy of Building Your Own.

I also wrote about Why I Still Blog — and Why the Future of Blogging Is Connected.

# More

Personal Websites
Personal Blogs
Future of Blogging
Creating your own website with Static Site Generators (SSG)
My approach to running a link blog
On 10 Years of Writing a Blog Nobody Reads | flow2
Writers are fleeing the Substack Tax
Open Web and Open Social with protocols such as AT Protocol for the win.
- Why websites? - from anna - pckt on top of AT Protocol

# Beautiful Websites

Beautiful Websites

Origin: Why does every personal website look like this now? - YouTube

Microsoft Fabric

Tue, 08 Sep 2026 13:22:15 +0000

New announcement 2023-05-23 about the single place of MS; it’s like a Databricks replica, except for the tools in Azure without needing to go into Azure (see more below). See the announcement Introducing Microsoft Fabric and Copilot in Microsoft Power BI.

They also announced the integration into git for some Power BI artifacts (RW Introduction to git integration). But it’s not there yet as they exclude most of the things. It will probably take a long time until they are ready, as usually, MS products have lots of visuals inside the files as well (same as in the SSAS builder where all is XML with lots of IDs)

# Features

Can cross-cloud data analytics be easy? | Microsoft Fabric - YouTube

These features are possible due to an open Data Lake Table Format like Delta Lake, which they packaged into MS OneLake, with more security and integration (lineage, etc.) on top of it. Similar to what Databricks is doing with notebooks and Lakehouse.

You can combine different Query languages and computes. I know they use spark under the hood to wrap up, but for T-SQL, they might use a serverless SQL server, or Analysis Services, depending on what you need.

# Comments Simon Whiteley Video

Some comments and notes:

it’s a sass service on top of Synapse, power Bi and more integrated with a new name, which is fabric
what is different is the communication: They talk about Fabric the same way they talk about MS Excel.
Spark in MS Synapse is now called Synapse Data Engineering

His comment: it looks all nice: but will it be seamlessly integrated?
- In synapse everything was stored in different formats, a little SQL, a little delta, etc. The fantastic news is, that in Fabric, everything is Delta Lake!
  - even if you write T-SQL and store the table, that will be a delta table.

There is no Azure, there is only SQL, Delta, and Spark. If logging into Fabric, you log into Power BI. It’s a software as a service, not AZURE. Wow!

# Lakehouse reference

Microsoft Fabric is basically a Lakehouse implementation by Microsoft. Others who have their own are Databricks with Delta, and Onehouse has theirs with Hudi calling it XTable. Dremio is also building one on Iceberg, and Snowflake is getting more open with Apache Iceberg.

Comparison to Databricks (User Experience)

This user got quite frustrated as of 2025-02-04 with the forced move to Fabric, coming from Databricks: Considering resigning because of Fabric : r/dataengineering.

# Declarative Metrics or Business Logic

I wonder, where is the business logic stored within Power BI, or is there a Semantic Layer? –> Tweet

Simon Whiteley:
Yeah, that’ll be your Power BI models most likely. You could have logical layers in either Lakehouse or Warehouse artifacts, but your metrics will be Power BI measures directly on top of the Delta Table objects.

Thomas Gremm on LinkedIn:

Mostly, in the transformations of the ETL/ELT jobs/dataflows/pipelines between the Delta tables, similar like in the Medallion Architecture known on the Databricks side.
The business logic spreads over certain products (Factory, Data Engineering, Synapse DWH etc.) in certain code (Spark, T-SQL, KQL)

You can also put business logic into PowerBI data models (like DAX measures, calculation groups etc.), but then it should be handled like a Data mart.

The truth should be persisted in the Delta tables (with Parquet as underlying), which is analogous to the Gold layer in the Medallion architecture in Databrick’s logic (=the well-structured business presentation layer).

Maybe Tabular Model Definition Language (TMDL) is any good, it’s human readable.

# Engine

The Data Lakehouse part of Fabric is basically also just Spark or Python and Delta. The Data Warehouse part has it’s own proprietary engine which is actually quite interesting as it allows T-SQL and multi-table transactions on top of Delta. Patrick Pichler on LinkedIn

# Use with AI

But in the age of AI, I am just pondering whether Fabric would be able to keep up with only focus on GUI when code-first tools have a better integration with AI. Especially, FDF’s JSON structure is so convoluted for hand editing and reviewing AI’s changes. You could say Co-pilot but it hardly feels good trying to get work done with it. Reddit: Perspective on Microsoft Fabric

# Release

2024-11-26: With the recent release of pure Python Notebooks in Microsoft Fabric, the excitement about these lightweight native engines has risen to a new high. Source
- Use Python experience on Notebook - Microsoft Fabric | Microsoft Learn

# Why not

From Is it just me, or is Microsoft Fabric overhyped? : r/dataengineering as of 2025-03-11:

No Local Development

There’s no way to run a local Fabric instance and connect it to an IDE.

Being forced to use the web UI for navigation is inefficient and unfriendly.

Poor Terraform Support

After 10 years of development, we’re still at step zero?

Terraform, which is standard for infrastructure as code in data engineering, has almost no meaningful support in Fabric.

Git Integration is Useless

While Git integration exists, what’s the point if I can’t develop locally?

Even worse, Azure Data Factory isn’t supported, which is a crucial tool for me.

No Proper Function Support

Am I really expected to run production pipelines in notebooks?

This seems like a recipe for disaster. How am I supposed to test, modularize, and run proper code reviews?

Notebooks are fine for testing, but they were never designed for running production ETL/ELT.

# History

Evolutionary History of Microsoft Fabric - Spreadsheets to Lakehouse - YouTube
? how much is VertiPaq still used?

# Further Reads

Microsoft Fabric Introduction | PPTX by James Serra
- with OneLake and Direct Lake/DirectQuery

Origin:
References: Power BI

Writing Manually (In Times of AI-generated Content)

Tue, 08 Sep 2026 08:46:59 +0000

In times of AI Writing, writing manually is more important than ever. As with many AI Generated texts, I’d rather see the prompts, it would have more soul and character than the output.

I couldn’t agree more with the sentiment of

I’d rather read the prompt by Clayton Ramsey (Id rather Read the Prompt)

Writing yourself, writing manually is much nicer, to hear your unfiltered thoughts, than condensing them through an LLM, and get average-sounding sentences with no soul. To me, LLM writing is soulless. I even started to turn to Grammarly and Copilot, as these were a mere distraction to the actual task at hand: writing. Instead of writing, I was constantly grammar fixing, and ultimately, nothing got done. Bsky

My experience is in writing, every time I use it for a bigger task (re-structure, or telling me the missing chapters), it does a very average and most importantly, distracting task for me. Instead of following my headspace, I am now adding some averaged chapters that can be found everywhere.

Also from Clayton’s article above:

I would rather see than the original prompt. The resulting output has less substance than the prompt and lacks any human vision in its creation. The whole point of making creative work is to share one’s own experience - if there’s no experience to share, why bother?

If it’s not worth writing, it’s not worth reading.

A great analogy found on Hackernews “Using an LLM to do schoolwork is like taking a forklift to the gym”.

If all we were interested in was moving the weights around, you’d be right to use a tool to help you. But we’re doing this work for the effect it will have on you. The reason a teacher asks you a question is not because they don’t know the answer." HN

# Like Driving A Car

Is writing manual like driving a car manually, instead of automatic? At first, I thought yes. But on the contrary, shifting automatic is a convenience that benefits only those who drive the car. The similarities are that both are faster, easier, and do an average job. However, the quality of automatic shifting is much superior to average AI-generated content. AI models are probabilities and are not made for writing. Yes they can write, but, they all sound the same, see How to detect AI Writing.

And with AI writing, we don’t do it for ourselves; we do it to share or impress others. If we only do it for ourselves, why wouldn’t we write it ourselves?

It’s not yes or no, but it’s a bit a nuanced difference. In shifting gears, you don’t need to have character or soul. If you are a race driver, you might shift to be much more efficient and better for each situation. If you go up, shift faster; if you go down, keep a low gear to make the motor break instead of breaking. So there’s some nuance, but it’s more simplicity whereas writing is Art, a way of communication. And much more than just shifting a gear. #thoughtoftoday

# Craftsmanship

Besides creating art manually, it’s more fun too. Craftsmanship will come back into vogue once we all identify AI Generated Images. It’s even more true for written words. The creator of OpenClaw just said that he is allergic to AI-written articles, but codes all day agentic.

So there is something to be said about human-made, not only Human in the Loop.

# Writing with an Asterisk

So very well explained, The AI Asterisk by Tom White.

Was it you? Who wrote it?

Without the machine would those small fragile thoughts still be nestled in the safety of your creative womb?

Maybe it’s neither extreme. Maybe the thing you made is some Ship of Theseus where AI’s planks got swapped in so gradually that nobody, including you, can point to where you end and the tool begins.

That’s the AI asterisk.

# Blogging: Addiction, and why Writing Manually

AI is very addictive, but also brings a lot of value as it can confuse us being productive versus just producing Shallow Work.

Blogging is also the antidote, if done manually. But not everyone can withstand the urge to use AI writing. Writing is hard at the end. So it’s much easier to take a shortcut. How do you withstand it? Have you come to the seduction of just having AI write it for you?

To me, it’s about the fun, and what I learn. Especially when I write connected as I explain in Future of Blogging and my essay on Why I Still Blog — and Why the Future of Blogging Is Connected.

# The Case for Writing Blogging

It helped me learn what I was doing better, and it helped me to coalesce my thoughts. Hamilton Carter

I couldn’t agree more with Hamilton. Even if AI were to get better than humans, still, for the sole benefit of learning, it’s worth doing. Plus, people who follow know it’s a human with character and quirks. More on why it’s important to have a sould and character at Will AI Replace Human Thinking?.

# Exhaustion

AI Makes me exhausted, that’s why I write manually still, and I’m convinced more every day. The less I write myself, the more I outsource my thinking, and therefore also my learning.

It’s less fun, and it makes me more exhausted. As you said, use it for research, but the storyline, the experience, the heart, the soul of your writing, should come from you, from real insight or a felt struggle.

# Why Developers Use LLMs to Write Blogs

Here’s a great survey to answer the question of Why Developers Use LLMs to Write Blog Posts.
Some quotes:

The “Always” group had the highest concentration of inexperienced writers. 40% reported they had never written blog posts previously, and another 20% said they wrote only rarely (twice a year or less).

Or:

“I have two writing ‘personas’: one for work and one for my own media outlets. I use LLMs more for my ‘work’ persona because it is better at corporate style content. For personal use, I only let it proofread my text and provide suggestions for specific cases (e.g., help with an expression).”

Or a suprising one:

“I mostly use AI to produce writing I know no one will read, but am required to write and am not given time to write to my own standard.”

Again, I’d say at the end, it’s a mix of many things, some of which we have discussed above, but read the full article with more comments and quotes. But also, Writing is Hard, still, and that’s why people do shortcuts, due to time or just no interest.

# How to Work with AI

Writing with AI, is skipping the thinking. My thinking happens during the act of writing. There’s a blocker, I can’t get around the structure, I change 3-10 times, I write a bit, I change again, I find something, I start a new intro, or start over.

In my opinion, if I skip this, which is very appealing to do, as it’s much easier to ask to create the outline, but this is also where the magic happens to me. Also, somewhere where I appreciate comments and from actual humans, as they give me better suggestions than the AI models.

The Voice-to-Text translation is nice in the car, but the real thinking comes when I sit down and need to put it into proper sentences. But yeah, there are so many ways, and everyone is different, so it’s best for everyone to explore. And yeah, just dwelling with the idea in the head, not just for an hour, but days, is when my subconscious does a lot of the hard work.

All of these is why I prefer Writing Manually.

# Generative Search Optimization (SEO but with AI)

Generative Search Optimization (GSO) is the new term for optimizing to appear in LLMs. But, there is also the approach of generating with AI to land on Google, rather than writing manually.

Charlotte Schmitt shares her skills, to optimize website pages automatically with AI. To me, it also depends on what you optimize for. I’ve gone through Charlotte’s mentioned experience of generating 100s of SEO pages with AI in Airbyte, and I never approved “AI-generated content”.

I guess this approach can work for historical pages where you compare X with Y, or where data is readily available online, where the information is purely informational.

But it will never work for human consumption, where people read for pleasure or want to learn something new. If Consumed by a Human, Should Be Written by Human. Because writing a story that is new, genuine, and has a character or conviction is not what AI can do or have. In the end, that is what people want to read in my opinion: a story with a soul, someone telling something. ^4f0d24

# Further Reads

Should I Change My Writing because of AI
Jordan Tigani on AI Writing Tools Limit Clarity of Thought ^6eb012
Rick Lewis Substack Note
- Organic human thought and expression is the equivalent to me of how tropical rain forests are called the lungs of the world. Bulldoze the rain forests and a few decades later we’re like, oh wait, those were really really important.
Seth Godin on Real artists…
Content Creation
Favorite Writer Pencil to write by hand
Why You Should Almost Never Use AI to Write Anything Substantive
The revolt of the reader | The Observation Deck

Origin: ChangeLog Podcast newsletter by Jerod Santo.
References: AI Writing

Keep AI Out of Your (Obsidian) Vault

Tue, 08 Sep 2026 08:46:43 +0000

Everyone is using Obsidian for AI, or wants to use it to become more productive. But I think it’s a dead end.

The reasons are clear: Obsidian notes are open on your local disk in an open format, Markdown, and accessible to everyone, including your AI agents. This makes it as great and easy an integration as possible. But should you actually use AI with your notes? And if so, what are the use cases or ways you should interact with it?

First of all, if you use Vibe Code Agents with Obsidian, use the Obsidian CLI. It will be much faster to return your files, search, or act, grepping through the files.

Second, I’m a strong proponent of avoiding adding lots of AI-generated summaries[^1] or other on-the-fly-generated text to my vault. The reason is simple: over time, I don’t know anymore whether the content was written by me or by an AI, and my own, much more valuable thoughts get diminished by “AI Slop”.

Also, when searching for something, you need to fight through the noise of generated stuff. If you only have your own writing, it’s all valuable, or at least there’s a reason why you noted it down. There was a conviction, idea, or something that moved you. ^90383c
Yes, it’s good in certain areas, and mostly on-the-spot summaries seem great, but every time I come back to them, I find them very average. It doesn’t help me, as it didn’t highlight the things I would, and I need to reread anyway.
What I do sometimes with Obsidian Webclipper, if I create a new note, is summarize it in one sentence or a few. I clearly mark it as AI-generated and even put it in a quote, so it’s clear to me, and to anyone in 5 years, that it wasn’t mine. If I have added my own thoughts and writing, I will just remove the generated paragraphs (not my words).

But, use it for advanced research, like finding related notes. I think this might be the best use case. But don’t use it for tagging or organization, because eventually all your relations and connections won’t count for anything, since they aren’t made by you. The power lies in the deliberately created graph of notes, your very own Second Brain.

My Obsidian Vault with Smart Lookup and related notes with Smart Connections and local graph | Also check out Vim Motions for Writers to see my writing workflow and how I write

I personally don’t use it like that yet, but I might one day with a powerful local model, since I have lots of sensitive notes I don’t want to upload anywhere. Plus, it takes away the thinking part of My Obsidian Note-Taking Workflow. I don’t want to replace my human thinking yet. But I use a mathematical or local model with Obsidian Smart Connections, or the Graph Analysis plugin.

I’m pretty sure that if you go all in with using AI to generate your notes, you will end up with less clarity and fewer insights. You might start fresh very soon, as you won’t get value from it, and stop maintaining or adding ideas.

[^1] To be clear, I mean long summaries, no harm in a 1-2 sentence intro in a note.

Long AI Summarization is usually just noise

Kepano says, too: “A summary of a PDF is noise. An insight I had from reading the PDF is signal”. Tweet. Read more Other Opinions by Jason Fried, Paul Graham, Ted Gioia, Mitchell Hashimoto, and many more.

# Search is an Organizational Question

I think if you have an urge to do something, do it in a separate vault, or just use a database, e.g. DuckDB, and do all the fun stuff with AI as I did for search and clustering. But don’t mix your precious notes and Zettelkasten for it. Not yet, at least, but probably never.

I have 25'979 files totaling 3.5 GB, all in a single vault, and I find everything in split seconds (using the Omnisearch plugin). Search is pretty much an organizational question; e.g., I use Zettelkasten, and some refer to Memex as its digital version.

But if you need more advanced features, you can just add the Obsidian Smart Connections plugin and get vector and similarity search, etc., out of the box.

# Your Notes Will Be Your Prompts Tomorrow

Remember, your notes will be prompts or libraries for AI tomorrow, but not if you generate them.

# Cultivating Knowledge

Also on the notion of cultivating knowledge for AI systems, I’d say I’m more interested in cultivating for myself, long-term 🙂.

I imagine if humans do not have their own knowledge bases, everyone will have the same (average) notes based on AI models, not 100% of course, but there is no conviction, no decisions made, just all equally similar.

This is another reason we also need human-curated knowledge on a large scale. AI models will retrain on this data, better than on synthetic, fake data, IMO, though we need to find an economic model that also works for the creator of these insights and knowledge.

# Whose Knowledge System is It?

Using Claude Code (or another agent) to generate your system and make connections, like , is not your system. If you want a tool for retrieval, yes, sure, but I have a second brain and a knowledge system where my thinking happens. I make the connections because I had an idea, or something told me to do so.

Great video on this with more of the same thinking I have at Obsidian vs Claude — Why I’m Not Worried About AI Killing Note-Taking Apps. He says elegantly that a second brain is a system; the connections and writing parts are not inefficient; it’s the process of thinking and learning.

Claude is just a tool, like Obsidian, but as File Over App, by the CEO of Obsidian, goes, it’s not about the app anyway, it’s about the files, your thoughts, your insights. So if it’s someone else’s, heck, even generated, it’s not why I use the second brain. Can be useful, too, sure, but not as much as my new ideas, the thinking I get from my knowledge system.
Touch some grass, they say.

# Examples

Some examples that identify this phenomenon quite well, or help us understand and think through it better.

# It’s Hard to Finish an Idea that is not Yours

While I use Claude a lot for my Omarchy plugins and Linux optimization/failure detection, I don’t use it for writing and note-taking, especially for coming up with unique ideas and driving them home by writing them through to the end.

It’s so hard to finish an idea that is not yours and is just suggested by AI.

# Andrej Karpathy’s LLM Knowledge

Andrej Karpathy LLM Knowledge Auto Research, which is another good example of how not to do it (IMO), especially if you like to learn. Sure, it’s fun and helpful, but it will lose its appeal and value over time. It’s like this Reddit quote says it well:

I definitely agree that the ‘Karpathy LLM Wiki’ mediated knowledge base everyone is excited about is not a useful way to actually learn or manage your learned and curated data for the reasons written. If instead you do just want an automated research summary of some knowledge - sure why not, but it’s not particularly interesting and I really don’t get the excitement about it. Reddit

Some great discussion in this thread on this topic.

# Does not Scale, or Will Die Very soon

Automating the links and connections like this won’t sustain for YOUR second brain, for your ideas and notes - you are skipping the thinking, and IMO, the insights. Most of my ideas come from linking and seeing the links I made myself, sometimes years back.

A good example of what not to do for your own vault. Tweet

For sure interesting to see what comes out, but I wouldn’t recommend it for your “second brain” where you develop your thinking and YOUR ideas.

If you want the above brain, which is not yours, you should use Wikipedia. Wikipedia graphs and wikilinks give you great insights, as the automated AI ones (probably more), but made by human collaborators. But if you want to create your own knowledge base, I’d do it manually. The payoff comes over the year, it compounds.

# PARA Method: Dedicated Folder

The PARA method by Tiago Forte is super convenient - and you could create an AI folder in resources and put all generated notes there and hide that from search or through other mechanisms. This way, you could still have AI-generated insights or notes that you can link, if helpful, to your original notes.

# Voice to Text

AI with Voice to Text can be another approach to use with Obsidian, which makes sense when you are in a car or so.

But because of my talking much more compared to when I write, and to AI or translator service not understanding all my words, it’s much more work for me than just typing it out. Plus, I usually skip the thinking, though sometimes talking it through can add to the process.

# Slowing Down: Have More Impact Long-term?

See Slowing Down.

# Creativity by Its Very Nature is Forward-looking

Creativity is Forward Looking

# Further Reads

Origin: Li by mehdio, Obsidian
References: Writing is Thinking!,

Writing with AI: What's the Best Approach?

Tue, 08 Sep 2026 08:44:30 +0000

To me, AI writing always results in this:

They always look nice at first sight, but over time, they look all the same: complex, unnatural, and busy. Same as DALL-E images.

And if it’s read by a human, it should also be written by a human. Nothing worse than having to read through a stack of elaborated text that has been generated on a small prompt. Naval elaborated it best at the The Differences between Writing and Coding

# Writing Styles: AI Extended (bloated), and the Prmpts, the Raw Writing

Based on the The Differences between Writing and Coding that Naval Ravikant shared, writing is consumed by human beings, and they discuss, that Id rather Read the Prompt, why would I need to read your fleshed-out article by AI, and not the lean prompt with all the insights? That’s what Naval asked Garry Tan, and they didn’t have an answer.

It’s like the below meme, AI making you Exhausted both on the creation time, as it never ends, people have more work than ever, but also on the reading side, as you need to read everyone’s bloated AI-written stuff, rather than just the prompt.

AI turns bullet points into a nice text - oh look, AI puts this blob of text into bullet points - Meme

Great example: Generating an email that we sent do an actual human being | Source: AI Written, AI Read cartoon - Marketoonist | Tom Fishburne

Also, Id rather Read the Prompt.

Same as we do in programming: sharing AGENTS.md or Agentic Skills so others can use that insight too, rather than sharing 10 different solutions. Obviously, as explained in the difference between writing in programming, it’s not the same as a computer consumes programming, and if you solve a problem, why not generate the app, the solution, and share that? We can’t do that in writing.

# The Two Kinds of Writing Styles with AI Models

What if we have two kinds of writing:

The elaborated but bloated (AI-enriched) one - like what Garry Tan is doing on Twitter or articles as he says
Or the raw prompt, with all the ideas and insights lean and easy to read (not the solutions, maybe?). Very much like the philosophy of Id rather Read the Prompt.

And as I argue in Manual Writing and Will AI replace Humans, the second point has character, a soul and human behind, no AI model will ever have.

# How Do I Write with AI?

I was wondering if AI should suggest to me writings - a question asked by Euwyn Goh . It might for brainstorming. I don’t want AI to write my sentences, but it could help me brainstorm.

Usually, the way I work is: first, I do brainstorming, maybe some research on what happened in history. In that phase, AI might help. But also, most often when I ask what I should add to my articles, AI says to add “real-world examples”. So, most of the time it’s not that creative. And the worst part, after using AI, I usually haven’t done my work, and I’m more exhausted than when I just write it 🤐.

What I used it for just recently, though, was to go through my notes, list MY notes, and add them to the chapter of my book (all in markdown, which’s why it works well). What’s nice is that you provide a way to train my own data for that AI using a local model. At least that’s what I’m trying a little at obsidian-note-taking-assistant with local first Second Brain RAG for any Obsidian Vault (works with any Markdown). Works with DuckDB. Also includes a web-app, see the blog article at Building an Obsidian RAG with DuckDB. Or more examples at Second Brain Assistant with Obsidian (NoteGPT).

# Alternative, Writing Manually

I’m all in for Writing Manually and not replacing my thinking with AI, especially if a human is consuming it.

# If Consumed by a Human, Should Be Written by Human

If Consumed by a Human, Should Be Written by Human

# AI Writing Tools

Byword
ChatGPT
LEX by Dan Shipper
Claude AI with Claude 3.5 being much better than ChatGPT 4.0
Spiral by Dan Shipper and Every (Writing Tool) ^114b88
River Docs: The AI Document Editor
Wordsmith by Euwyn Goh

# Results of AI Written Analyis

# Pangram Report per 2026-07-09

AI Content Is Everywhere on Social Media, Especially LinkedIn | Pangram Labs - obviously dedecting AI content is also not reliable, again, AI is not deterministic, it might even say once it’s AI written and once not if you asked it twice, but here are some reports to ponder from that article:

And LinkedIn and Medium being top, is what I saw as well, no test needed. Substack is very low makes sense too, as there’s no API or no Social Media Scheduling Tools and Platforms yet to publish notes or substacks newsletters, plus it’s a platform for actual writers.

Here are two more overviews:

Source

Reddit replies are rarely AI written - though I know that is also depenend on the subreddit for sure:

Source

How the data was collected

From the article:
“Collectively, since the Chrome Extension’s launch on April 24th 2026, users who opted into sharing their data for research helped us create a dataset of 1,002,627 posts across several of the largest social media platforms on the internet: LinkedIn, Medium, Substack, X/Twitter, and Reddit.

Each post in our dataset is counted only once, and we only scan items that are longer than 50 words. Every post was analyzed with Pangram 3.3, our latest AI detection model, which achieves a 0.01% false positive rate. This dataset allows us a direct window into what AI-generated content people are seeing on their feeds at this point in time.”

# Further Reads

Course with Dan Shipper: AI writing gets a bad rap.
Plagiarism: Check if AI Written: Tools to Check Plagiarism and AI-Generated.
Counter Arguments to write with AI:
- Technical writing is too important to leave to language models
- Will AI replace Humans
Don’t be a meat proxy
Opinion | I’m Begging You: Never Write With A.I. - The New York Times

Origin: AOMW Course- The Art of Modern Writing (Foster.co), AI Writing with Dan Shipper - YouTube
References: ChatGPT, Creative Prompt Engineering , My Services and Target Audience (SSP Data),Brainstorming Ideas for Earning Money - Book, SSP Data

How to Get Started with Data Engineering

Tue, 08 Sep 2026 08:15:32 +0000

Whether you work in a different field or are curious about what Data Engineering is, this is the right place to start as a data engineer.

First, you need to get a short overview of What is Data Engineering, and then read the trilogy of the defined articles that started the field by Maxime, starting with The Rise of the Data Engineer.

Then it really depends on what your role as a data engineer is. Are you doing more platform work, more Business Intelligence work, or Data Analytics, or even Software Engineering? All of these do slightly different work and require different skills.

The typical toolbox of a data engineer can be found Data Engineering Toolkit (including a 3-weeks roadmap), and the most important concepts are at Data Engineering Concepts:

If you prefer a top-down approachthat introduces you to the field, the history of data engineering, and its challenges, check out my book, with the chapter Introduction to the Field of Data Engineering, and continue with the rest of the chapters.
If you like a bottom-up approachand already have some knowledge, but want to learn more and see where your curiosity leads you, definitely check out my Data Engineering Vault as part of my Second Brain. You can search and click through backlinks and outgoing links, or use the interactive graph provided that links to the relevant blog articles or book chapters.
Of if you like hands-on projects, check out the Open-Source Data Engineering Projects.

# Should You Still Get into Data Engineering in 2026?

I think if you don’t like being the center of attention, prefer working in the background, enjoy problem-solving all day, want to keep up with the latest, and like data, then it’s still a very good path.

Surely it’s hard to say what happens with AI, so maybe instead of focusing too much on learning as many programming languages (still learn one programming language) or details, focus on the principles, the Data Engineering Fundamentals.

# Community and More

For more links to start, you can find some of my maps of content notes, which provide you with many more links and other platforms to learn from, with curated, up-to-date lists for each category:

Books of Data Engineering: A selection of essential reads for every data engineer.
People of Data Engineering: Learn from the pioneers and current leaders shaping the data engineering landscape.
Data Engineering Glossaries & Handbooks: Glossaries and Handbooks that explain the complex terms of DE.
RSS feeds for Data Engineering: My list of best data engineering blog posts as RSS feeds.
Data Engineering Whitepapers: Whitepapers that define the foundation of data engineering.
Data Engineering Blogs and Newsletters: Providing insightful articles, podcasts, and newsletters on DE.
Data Engineering YouTube: Popular YouTube Channels focusing on DE.
Learning Data Engineering: With more resources, such as hands-on projects, courses, and boot camps, to start learning.

I hope that helps. If anything is missing or you’d like to know more, reach out or follow along at my newsletter. Subscribe directly here.

Origin: Data Engineering

Data Engineering Fundamentals

Tue, 08 Sep 2026 06:57:48 +0000

To me, the fundamentals of data engineering are well described in Fundamentals of Data Engineering Book with the Data Engineering Lifecycle:

Image from the above mentioned book | Source

If you want to learn related fundamentals, look at Data Engineering Toolkit or Data Engineering Concepts, where I list the most important toolkits for a data engineer and the concepts we need to deal with throughout our career.

# Further Reads

More at:

Learn the fundamentals as in my book: 📖 Patterns of Data Engineering
Or find the Data Engineering Vault with 1000 interconnected terms.

Origin: Data Engineering- Trends and Predictions

Data Engineering Lifecycle

Tue, 08 Sep 2026 06:51:43 +0000

In today’s landscape, a data engineer is pivotal in overseeing the entire data engineering process. This involves gathering data from diverse sources and ensuring its availability for downstream applications. A deep understanding of the various stages in the data engineering lifecycle is essential. Additionally, a data engineer must possess the skill to evaluate data tools effectively, considering various aspects such as cost, speed, flexibility, scalability, user-friendliness, reusability, and interoperability.

Illustration of the data engineering lifecycle, from Fundamentals of Data Engineering

# Data Lifecylce

Another perspective can be seen in this Tweet where we analyse the Data Lifecycle:

Read more on the data lifecycle in my book chapter about Challenges in Data Engineering.

Or related insights, see Data Engineering Architecture, such as the one from A16z.

Case Study: Open Data Stack Project

The Open Data Stack project exemplifies practical application, incorporating key lifecycle components like ingestion, transformation, analytics, and machine learning.

Further reading: The Evolution of The Data Engineer: Past, Present & Future.

# Undercurrents

These are the foundational elements of the lifecycle, pervasive throughout its various stages: security, data management, DataOps, data architecture, orchestration, and software engineering. The lifecycle cannot function effectively without these integral undercurrents.

# Core Principles and Links

Here are the above core principles of the engineering lifecycle, added with my own thoughts or features.

Data Integration (Ingestion)
Transformation
- Semantic Layer / Metrics Layer
- Physical transformation (e.g., dbt)
Storage Layer
Analytics and Machine Learning
Additional Elements:
- Data Catalog
- Reverse ETL
General Foundations (Undercurrents):

# Data Lifecycle

Related is the Data Lifecycle and Data Canvas.

# Let’s not repeat ourselves

With the hype cycle, we have a tendency to repeat ourselves with ever-new tech.

But let’s integrate new data tech into the engineering lifecycle instead of creating new siloed work.

The picture below illustrates, with the chasm hype cycle, the engineering behavior is to skip fundamentals, adopting ever-new tools instead of sustaining architectural patterns that work.

graph LR
    subgraph "Engineering Behavior"
        P1[Problem Discovery] -->|"Search for Quick Solution"| P2[Build/Adopt New Tool]
        P2 -->|"Technical Debt Accumulates"| P3[Maintenance Challenges]
        P3 -->|"Research Existing Solutions"| P4[Discovery of Established Patterns]
        P4 -->|"Integration & Optimization"| P5[Sustainable Architecture]
        
        P6[NIH Syndrome] -.->|"Not Invented Here"| P2
        P7[Learning Curve Avoidance] -.->|"Skip Fundamentals"| P2
    end
    
    
    classDef vectorTech fill:#e1f5fe,stroke:#0277bd,stroke-width:1px
    classDef engBehavior fill:#e8f5e9,stroke:#2e7d32,stroke-width:1px
    classDef convergent fill:#fff3e0,stroke:#e65100,stroke-width:1px
    classDef connection stroke:#999,stroke-width:1px,stroke-dasharray: 5 5
    classDef convergentLine stroke:#e65100,stroke-width:2px
    
    class V2,V3,V6 vectorTech
    class P1,P2,P3,P4,P5,P6,P7 engBehavior
    class C1,C2,C3,C4,C5,C6 convergent

# Further Reads

Learn the fundamentals as in my book: 📖 Patterns of Data Engineering
Or find the Data Engineering Vault with 1000 interconnected terms.

Origin:

Home Made Ramen: Step-by-Step Instructions

Mon, 07 Sep 2026 21:45:01 +0000

This is a delicous Ramen soup, that you can make at home. It takes some time, but it’s fully worth it.

Also be sure, to go to an asian store near by to find all authentic incrediantes and flavours.

# Soup Broth Ingredients

Before making the full ramen, we need to make the heart of Ramen, the broth.

Here’s the incredients for that:

8 higher-welfare chicken wings
1 handful of pork bones, (ask your butcher)
750 g higher-welfare pork belly, (skin removed and reserved)
2 thumb-sized pieces of ginger
5 cloves of garlic
1 heaped tablespoon miso paste
500 g dried soba or ramen noodles
8 small handfuls of beansprouts, (ready to eat)
8 spring onions
1–2 fresh red chilies (depends… without is better)

# Ramen Instructions

All instructions including picutures of how it looks.

You need to clean the meat. Therefore boil it until the brown and blood goes out:
Throw the water away and use new one
Put the meat and all ingredients inside the bowl and put the new water:
Use sake as well (only a little bit):
When it is cooking, take out the bubbles (this is still from the meat). We want clean soup broth:
Prepare the tari-sauce that is used for the pork belly later (put all ingredients like two garlic (squeezed), ginger slices and soya sauce and vinegar and cook it for 5 minutes. Turn off and leave it for later:
Take a nice pork belly, cut it and put it in the already cooking ramen:
Soak the meat with the ramen and cook it for 1h
The rest of the ramen, let it cook for 3 hours
The pork belly should be around 1h, then take it out and put it in the tari sauce
Put it in the fridge until you are ready to eat the ramen

Next day or when you ready to make the ramen:

Reheat the soup, cook it on high level to get a more concentrated broth
Then when you have the right amount of soup/ramen, put 3 pieces of miso (if you have paste, that should already be in on day one with the soup)
Ready the ramen or udon noodles
Don’t need to be cooked, can be added to the hot soup and let it warm up
Slice the pork belly from the fridge
No need to heat up either
Filter the ramen so that you have the plain ramen:
Slice the soy eggs (see Ramen - Soy Sauce Eggs on how to make)
Last step, prepare the ramen
Put spring onions, bean sprouts, eggs. Soak everything with the very hot ramen, so that the meat, egg and noodles are warm
Enjoy! ;-)

# The Result 😋

Another (quicker) one:

# Pork Belly - the Ramen way

Note in above steps we did not wrap the pork like a real ramen. It was the easy variant. If you want the hard way, this is from another time we did it correctly:

and then you fry it before serving:

And the result with Tofu:

# Gallery of More Ramen

# Further Reads

Ramen Lord E-Book
Aromatic Oil to add to each bowl for Ramen

Origin: docs.google.com/document/d/1R722PsYbi4nz0ZdKJuBx-k2s1ZxLbop4UueuMPit21E/edit?usp=drivesdk, Steaming Ramen | Chicken Recipes | Jamie Oliver
References: Ramen
Created

Apache Arrow: The Foundation for In-Memory Storage

Mon, 07 Sep 2026 14:13:10 +0000

Apache Arrow is in-memory analytics format. It serves as a robust development platform for in-memory analytics. It encompasses a suite of technologies designed to expedite the processing and transfer of big data.

# Exploring the Arrow Libraries

Within the Arrow reference libraries, you’ll find an array of distinct software components:

Columnar vectors and table-like containers (akin to data frames), supporting both flat and nested types.
A rapid, language-neutral metadata messaging layer, leveraging Google’s Flatbuffers library.
Reference-counted off-heap buffer memory management, facilitating zero-copy memory sharing and adept handling of memory-mapped files.
IO interfaces for seamless interaction with both local and remote filesystems.
Self-describing binary wire formats, both streaming and batch/file-like, for efficient remote procedure calls (RPC) and interprocess communication (IPC).
Integration tests to ensure binary compatibility across implementations (e.g., transmitting data from Java to C++).
Capabilities for converting to and from other in-memory data formats.
Readers and writers compatible with various prevalent file formats, including Parquet and CSV.

# Visual Comparisons: Before and After

This project mirrors the objectives of Substrait Cross-Language Serialization for Relational Algebra.

# Ecosystem

Arrow: Defines a language-agnostic, in-memory columnar format where data can be shared between systems without serialization (no costly conversions like JSON or CSV)
Apache Arrow Flight (Protocol): A high-performance RPC framework built on gRPC to transfer ‘Arrow’ data between client and server at wire speed
Arrow Flight SQL: A protocol that extends Flight to send and execute SQL queries
ADBC: provides a standardized API for database interactions, making it easier for developers to query and work with databases using Arrow-native data (with/without Flight SQL)

Source

# History

Apache Arrow was announced by The Apache Software Foundation on February 17, 2016, with development led by a coalition of developers from other open source data analytics projects. The initial codebase and Java library was seeded by code from Apache Drill.

# Versioning Insights

# V6.0.0

Source: RW Apache Arrow High-Performance Columnar Data Framework

# Who is Using Apache Arrow?

Datafusion
ROAPI
ADBC
Blaze
Polars
ConnectorX
Cube Store
Daft
Dremio
LanceDB
OmniGraph
SedonaDB
Vaex
Arroyo
Spark Connect
Feather File Format used in Open File Format
Xorq
Streamling: Columnar streaming with native-speed plugins by Yaroslav Tkachenko

# Further Reads

Data Wants to Be Free: Fast Data Exchange with Apache Arrow | Apache Arrow
PyArrow for Large Dataset Processing - Confessions of a Data Guy
Zero Copy Architecture - Anurag’s Homepage
Book by Matt Topol: In-Memory Analytics with Apache Arrow: Perform fast and efficient data analytics on both flat and hierarchical structured data
Data Engineering Whitepapers
- Data Warehousing: Dremel: Interactive Analysis of Web-Scale Datasets
- Dremel Encoding: Dremel: Interactive Analysis of Web-Scale Datasets (Year 2011) used in Apache Parquet, and Apache Drill (which was seeded into Apache Arrow).

ROAPI
similar engine, but close-source, VertiPaq, see also SQL Query Engine

Modern Data Stack

Mon, 07 Sep 2026 09:48:23 +0000

The Modern Data Stack (MDS) comprises a suite of open-source tools designed for end-to-end analytics. This includes data ingestion, transformation, machine learning, and integration into a columnar data warehouse or lake solution, all complemented by an analytics BI dashboard backend. The stack’s versatility allows extensions for data quality, data cataloging, and more.

MDS aims to enable data insights using the best-suited tools for each process. It’s worth noting that “Modern Data Stack” is a relatively new term, with its definition still evolving.

Synonym Names

A burgeoning term, ngods (new generation open-source data stack), has emerged. Previously, I’ve referred to this concept as the Open Data Stack Project. Additionally, Dagster introduced the term DataStack 2.0 in a recent blog post. Open Data Stack is my own definition of it.

Closed Source vs Open Source

Closed Source examples: dbt, Looker, Snowflake, Fivetran, Hightouch, Census
Open Source alternatives: airbyte, dbt, dagster, Superset, Reverse-ETL?

Modern Data Stack on a Laptop

DuckDB: Modern Data Stack in a Box

# Why the Modern Data Stack?

A perspective from Reddit highlights the shift in data warehousing and analytics. It underscores the reduced need for extensive teams and infrastructure, thanks to new tools that streamline data management and reporting. Particularly for small and mid-sized companies, MDS offers a competitive edge in data handling, allowing even a single data engineer to manage vast datasets efficiently.

A notable article discussing Lakehouse, Metrics Layer, and Clickhouse:
The Next Cloud Data Platform | Greylock

# Integrating with Dagster

The downside of MDS is the unbundling of Bundling vs Unbundling, but Dagster helps integrate the full data stack together:

Dagster elevates the Modern Data Stack:

Explore more about its power with Dagster and Data Assets.

I often ponder over the ideal tools for a data stack. My preference leans toward a Cloud Data Warehouse such as Firebolt, Snowflake, BigQuery, Redshift, or Synapse, as a starting point.

The journey typically begins with Airbyte for data integration, followed by SQL-based transformation with dbt. Orchestrating the processes in Python with tools like dagster is crucial.

From there, I would integrate additional open-source tools based on specific needs: Spark for processing, Delta Lake for data lake formatting and ACID Transactions, Amundsen for data cataloging, and Great Expectation for data quality, among others. For smaller projects, DuckDB is suitable for local OLAP scenarios, while Kubernetes and DevOps provide scalability.

For teams without data engineering resources, closed-source options like Ascend or Foundry are viable alternatives.

Feel free to reach out for further discussion or clarifications.

# Counter Arguments against Modern Data Stack

# Modern Data Stack Alternatives

Some say PRQL as a DuckDB Extension and Dagster only could replace the whole MDS stack:

I’m with you at least 2/3 of the way. My preferred stack is PRQL + DuckDB + Dagster. I evaluated the space for work at my current company (was originally only DE, handling ingests from ~300 sources across various systems, on order of ~1k downstream tables in dbt + hundreds of dashboards + a handful of business-critical data+app features; now leading a small team).

I came away ranking dagster first, prefect second, everything else not close. IMO dagster wins fundamentally for data engineers bc it picks the right core abstraction (software defined assets) and builds everything else around that. Prefect for me is best for general non-data-specfic orchestration as a nearly transparent layer around existing scripts. PRQL as a DuckDB Extension | Hacker News

# The Death of Modern Data Stack?

Everyone is talking about the death of the Modern Data Stack, starting in 2026 and earlier.

The modern data stack is not dead anytime soon, or anything like it. Let’s call it Open Data Stack, or I don’t mind the name, but compared to when I started my career, I don’t want to give any of the open data stack tools away and do it myself.

# Other Data Stacks

Open Data Stack
Data-First Data Stack
Traditional Data Stack

# Further Links

The next layer of the modern data stack
What Is the Modern Data Stack? | Fivetran
Learning MDS with simulator as a cookie clicker 🍪: Data Stack Ada (by Christophe Blefari)
The Shape and Feel of the Post-AI Data Stack — Ian Macomber (Built on mostly Closed-Source, and Paid services)

Origin: Ask Simon | astorik
References: Cloud Data Warehouses Data Orchestrators Dagster Amundsen Data Catalog Python dbt Superset Metrics Layer Kubernetes Use closed-source if you don’t have the developers or the time Ascend.io Palantir Foundry Wiki

DataFusion: Fundamental OSS Query Execution Framework

Mon, 07 Sep 2026 06:51:14 +0000

DataFusion is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.

DataFusion supports both an SQL and a DataFrame API for building logical query plans as well as a query optimizer and execution engine capable of parallel execution against partitioned data sources (CSV and Parquet) using threads.

DataFusion also supports distributed query execution via the Ballista (Arrow) crate.

It doesn’t have its own storage format. That makes it ideal for running SQL queries on files in CSV, Parquet, JSON, Arrow, and Avro formats, whether they’re stored locally, in cloud storage, or on HTTP servers, similar to DuckDB.

# Use Cases

Data Fusion is a single node, not distributed (why it was donated to Apache Arrow as an in-memory technology?). If you want to distribute it, you can use Ray for Python as Andy Grove said in Andy Grove, or Ballista (Arrow).

According to Denny, it’s so fast, that you almost do not need distribution, but you can easily get it with Ray. Also as e.g. delta-rs is built in Rust, it’s so tiny, that he can use a Lambda function with it, which makes it extremely powerful. Imagine the difference to the Java JVM version, which comes with so much overhead and one reason Spark is so complex I guess.

From the GitHub Repo:
DataFusion can be used without modification as an embedded SQL engine or can be customized and used as a foundation for building new systems. Here are some examples of systems built using DataFusion:

Specialized Analytical Database systems such as CeresDB and more general spark like systems such a Ballista (Arrow)
New query language engines such as prql-query(PRQL) and accelerators such as VegaFusion
Research platform for new Database Systems, such as Flock
SQL support to another library, such as dask sql
Streaming data platforms such as Synnada
Tools for reading / sorting / transcoding Parquet, CSV, AVRO, and JSON files such as qv
A faster Spark runtime replacement (blaze-rs)

By using DataFusion, the projects are freed to focus on their specific features, and avoid reimplementing general (but still necessary) features such as an expression representation, standard optimizations, execution plans, file format support, etc.

# TableProviders

Integrate datafusion in dlt or similar, so we could do select * from slack.users.

Source YT by Jake Thomas

Docs for TableProvider:
Custom Table Provider — Apache DataFusion documentation

Custom Table Provider
Like other areas of DataFusion, you extend DataFusion’s functionality by implementing a trait. The TableProvider and associated traits, have methods that allow you to implement a custom table provider, i.e. use DataFusion’s other functionality with your custom data source.

This section will also touch on how to have DataFusion use the new TableProvider implementation.

Table Provider and Scan
The scan method on the TableProvider is likely its most important. It returns an ExecutionPlan that DataFusion will use to read the actual data during execution of the query.

# Why DataFusion?

High Performance: Leveraging Rust and Arrow’s memory model, DataFusion achieves very high performance
Easy to Connect: Being part of the Apache Arrow ecosystem (Arrow, Apache Parquet and Flight), DataFusion works well with the rest of the big data ecosystem
Easy to Embed: Allowing extension at almost any point in its design, DataFusion can be tailored for your specific usecase
High Quality: Extensively tested, both by itself and with the rest of the Arrow ecosystem, DataFusion can be used as the foundation for production systems.

# Comparisons with other Projects

Here is a comparison with similar projects that may help understand when DataFusion might be be suitable and unsuitable for your needs:

DuckDB is an open source, in process analytic database. Like DataFusion, it supports very fast execution, both from its custom file format and directly from parquet files. Unlike DataFusion, it is written in C/C++ and it is primarily used directly by users as a serverless database and query system rather than as a library for building such database systems.
Polars: Polars is one of the fastest DataFrame libraries at the time of writing. Like DataFusion, it is also written in Rust and uses the Apache Arrow memory model, but unlike DataFusion it does not provide SQL nor as many extension points.
Facebook Velox is an execution engine. Like DataFusion, Velox aims to provide a reusable foundation for building database-like systems. Unlike DataFusion, it is written in C/C++ and does not include a SQL frontend or planning /optimization framework.
DataBend is a complete database system. Like DataFusion it is also written in Rust and utilizes the Apache Arrow memory model, but unlike DataFusion it targets end-users rather than developers of other database systems.

# DuckDB vs. Datafusion

Comparing it to DuckDB: Duckdb Vs. Datafusion - by Matt Martin - Matt’s Substack

# Who is Using DataFusion?

Ballista (Arrow)
ROAPI: Build directly on apache arrow and DataFusion.
Cube with Cube Store
InfluxDB
GlareDB,
Lance

# Apple and DataFusion

Apache DafaFusion Comet

# Extensions

# Dbt (ADBC)

Since 2026-06-09 there’s a dbc connector, Ian shares:

This ADBC driver embeds DataFusion and makes it easy to use from 10+ languages through ADBC’s simple interface. The SQL syntax matches what you’d use in the DataFusion CLI. The driver takes no connection arguments; you control it entirely through the SQL you run.

Install with:

`1`	`dbc install datafusion`

# Releases

# V54.0.0 May 2026

The release also brings substantial improvements across join execution, table scans & query planning, continuing the project’s focus on building a fast and capable query engine says Dipankar Mazumdar on LinkedIn:

LATERAL joins: Enables subqueries in the FROM clause to reference columns from preceding tables, unlocking more expressive correlated queries and set-returning functions

SQL lambda functions: Introduces inline lambda expressions and higher-order array operations like array_transform, array_filter, and array_any_match for per-element processing directly in SQL.

Content-defined chunking (CDC) for Parquet: Aligns page boundaries with data patterns instead of fixed row counts, improving deduplication efficiency and incremental storage.

New Avro reader - Replaces the legacy implementation with the shared arrow-avro crate, delivering faster performance and better alignment with the Apache Arrow ecosystem.

This release includes roughly 11 weeks of development, 740 commits & contributions from 139 developers across the community. Full release: Release DataFusion `54.0.0` (Apr 2026 / May 2026) · Issue #21080 · apache/datafusion

# Further Reads

Why Startups Are Betting Everything on Apache DataFusion - The New Stack
Duck Hunt: moving Bauplan from DuckDB to DataFusion: LinkedIn by Duck Hunt
Algorithms on billion-scale graph using 10GB RAM: I love DataFusion! | Sem Sinchenko

Origin:
References: ROAPI Apache Arrow Rust

Constraints (for Life)

Sun, 06 Sep 2026 16:49:50 +0000

I like constraints to nourish artistic work, to unblock you when you have total freedom. I use them to enhance my writing or other types of work I do.

# The Problem of Total Freedom

Derek Sivers once said:

When you have infinite options, you often end up writing predictable or dishonest music.

The solution: make restrictive, arbitrary rules. E.g., you might have a Writer’s Block for writing an article, but if you set the rule of “only write in first form, ‘I’”, and only “about experiences you had”, and all of a sudden the ideas will come out of you like nonstop

# Constraints Result in Intuitive Tools

I believe Constraints make intuitive apps or tools, e.g., for data-specific. Rill Developer is a intuitive BI Tool, because it has constraints to the customization.

# Constraints Simplify Life

In Why Are We Here on Earth I wrote:

The Ten Commandments of the bible are like guidance in life. Like principles, people need Constraints. Like in things doing in life, and in life it self. An artist who has no constraints has a hard time creating anything. It’s like unblocking himself. Ten Commandments are the constraints of life, the law for how to live life before there were any laws in place in this world.

Origin: Initially from Derek Sivers, Experiments in music and life | Derek Sivers

Yazi

Sun, 06 Sep 2026 08:06:01 +0000

Yazi is a blazing-fast terminal file manager to replace GUI-first file managers.

For me, it’s a better Ranger that I used for years, built in Rust. Nice: it shows images out of the box, is super fast, supports Zoxide with Z, and can preview large files with DuckDB.

# Keybindings

Keybindings I most use.

# Most Importan

z for zoxide
Z for fuzzy search
s search by file name (fd)
S search by content (rg)
~ or F1 for help

# Default

See all in yazi/yazi-config/preset/keymap.toml. Find GitHub with all the information.

# Extensions

# Install DuckDB Extentsion

DuckDB to preview parquet, json, CSV etc.

# Install

https://github.com/wylie102/duckdb.yazi/tree/main?tab=readme-ov-file#installation

`1`	`ya pkg add wylie102/duckdb`

# What is it

Yazi plugin that uses DuckDB to preview data files

TL;DR: duckdb.yazi is a Yazi plugin leveraging DuckDB for rapid data file preview and summarization, enhancing file exploration within the Yazi file manager in the terminal.

For example, with duckdb.yazi plugin, you can instant preview .csv, .json, .parquet, and .tsv files directly within the Yazi file manager. Yazi, in case you haven’t heard, is a Ranger alternative, a fast terminal file manager written in Rust based on async I/O. It allows us to show images and now also data set files that we can explore immediately.

# Alternative

GitHub - yorukot/superfile: Pretty fancy and modern terminal file manager

# Further Reads

Origin:

Git for Data

Sun, 06 Sep 2026 07:13:04 +0000

Git like versioning on top of object storage/Data Lakes and databases.

# What are the benefits?

See my article on Branch, Test, Deploy: A Git-Inspired Approach for Data and Part 2, Git for Data Applied: Comparing Git-like Tools That Separate Metadata from Data.

# Tools

Tools to build git for data, either on object storage with plain files or inside a database.

# Data Lake

LakeFS
- Version control principles applied to data: Branching, merging, and rollback strategies
- Implementation options: LakeFS, Quilt, DVC, or native open table format capabilities
- Impact on data engineering workflows: How version control transforms development and testing
- Data Version Control · DVC (only for ML) -> Acquired by LakeFS (Data Engineering Acquisitions)
Bauplan (uses Nessie, at least according to their Data Engineering Whitepapers)
- The programmable Data Lake. Code-native platform for versioned pipelines on object storage with zero infrastructure management. Simple for developers, robust for systems. The data layer for AI systems.
- Pythonic-based
Nessie by Dremio
Lix: An embeddable change control system that enables Git-like features for any file format.
Lance: Branching and Shallow Cloning in Lance: Towards a “Git for AI Data”
Apache Iceberg itself has a branching feature: Branching and Tagging - Apache Iceberg™
OmniGraph: Company Brain

# Databases

Dolt: Dolt is a SQL database that you can fork, clone, branch, merge, push and pull just like a Git repository.
- They added DoltLite, a SQLite version controlled DB.
zero-copy / snapshot approaches: modern data warehouses (Snowflake’s zero-copy cloning, BigQuery snapshots) support this pattern.
Tigris: Fork Buckets Like You Fork Code | Tigris Object Storage
Supabase: The core approach is full instance branching. Each branch is a completely isolated Postgres database with the entire Supabase stack (Auth, Storage, Realtime, Edge Functions).
Tigris: The core approach is immutable object versioning with instant bucket forking powered by FoundationDB.
Guepard: Guepard Instant Database Branching
Turso: Supports Branching

# GeoSpatial

GitHub - koordinates/kart: Distributed version-control for geospatial and tabular data

# Potential Contender

xet-core by Hugging Face: Xet is on the Hub
SQLMesh with Virtual Data Environments (great share by Denis Arnaud)

# Research

# XetData

Data Engineering Whitepapers

Reproducible data science over data lakes: Replayable data pipelines with Bauplan and Nessie ^a13d97

Git for Data Paper by xet-core: Proposes a system that extends Git to efficiently handle terabyte-scale machine learning datasets through content-defined chunking and deduplication.

# Git-inspired Permute-Partition-Compress strategy

Comment by Antonio Boffa on LinkedIn:

The spectrum from full copies to metadata/catalog-based branching is very clear. It instantly reminded me of “Git is for Data” (CIDR'23 Data Engineering Whitepapers).

I’ve been looking at a related angle on the storage side in “On the compressibility of large-scale source code datasets” (Information Systems ‘25 https://www.sciencedirect.com/science/article/pii/S0164121225000974), where we use a Git-inspired Permute-Partition-Compress strategy to squeeze huge, versioned source-code collections (Software Heritage) while keeping them manageable. Check more on RW On the compressibility of large-scale source code datasets.

# Bauplan

Data Engineering Whitepapers shows how Bauplan does it with Nessie:

Source is the DE Whitepaper above from Bauplan on Reproducible data science over data lakes.

Ciro Greco, founder of Bauplan also says:

Engineers care about two things in production:

They want downstream consumers never to see corrupted or inconsistent data

And they want to avoid debugging failures under pressure.

His solution is:

To achieve this, a data system must prevent jobs from writing directly to production tables. Every run needs to execute on an isolated branch where all transformations, schema changes, and ingestions occur. If a job fails, the branch must remain open, with the exact failing state preserved, so the issue can be inspected without touching production. See comment on LI

# Whitepapers

See DE Whitepapers for git for data:

Data Engineering Whitepapers
Git for Data is version control concepts applied to Datasets and data pipelines.

Reproducible data science over data lakes: Replayable data pipelines with Bauplan and Nessie

Git for Data Paper by xet-core: Proposes a system that extends Git to efficiently handle terabyte-scale machine learning datasets through content-defined chunking and deduplication.

Building a Correct-by-Design Lakehouse: Bauplan’s paper on Git for Data, and building Lakehouse. It contains Data Contracts, Versioning, and Transactional Pipelines for Humans and Agents

GitLake: Git-for-data for the agentic lakehouse by Bauplan: The system lifts single-table Iceberg snapshots into lakehousewide commits, branches, and merges, letting agents work on isolated branches while humans review and publish changes. Pipelines run on temporary branches and publish through a final merge, so all outputs become visible atomically or none do.

# Further Reads

Origin: LakeFS
References: The State of DevOps in Data Engineering, Open Table Formats, git

Writing is Hard; and Meaningless with no Experience

Thu, 03 Sep 2026 11:55:56 +0000

Everyone who can write has a competitive advantage because not everyone can do it. Tim Ferriss said that when he felt competitive in angel investing or podcasting, he was thinking of going back to writing.

In podcasting, with money, you can bring in more money and better production quality. Writing on the other side, is hard. Harder to replicate.

After all, good writing is hard; exceptional writing may be the hardest there is.

# Is AI Solving This?

Big ideas are less about creativity and more about conviction. [..] So, what happened? ‘Sauce and other shit’ got incredibly cheap! [..] There is no AI prompt for conviction. Harry Dry

True writing is hard—and always will be. To me, you are writing from the heart and soul, and that’s a beautiful thing. More so these AI days. Writing from The heart.

Will AI replace Humans? I don’t think so. There’s no character, no soul, and it’s just average. Sure, it’s good at writing, but it’s not good at having a style, writing new things, or having a hugely large context (e.g., when I write an article over two years long, putting in my heart and soul). These writings will always have their place. We have digital print for a long time, yet books are still a thing.

As Harry says, AI has no conviction; There’s something missing, something human. Or picture this: who would like to read a book and one they liked momentarily, but then you want more and you find out that it’s written by a machine? How would you feel?

# Writing is a Wicked Problem

Writing is a Wicked problem. The Safest Job from AI may be Writing

Love it. To me, writing is also an art form, a craftsmanship, and only with experience do the written words mean something. Everything is meaningless if it’s just for the sake of producing text; if there’s no story, no hardship.

Also:
Large Language Models (LLMs)

# Further Reads

Origin: Tim Ferriss
References: Writing, AI Writing

Terminal Multiplexer

Mon, 31 Aug 2026 16:57:16 +0000

Tmux as the most know multiplexer of this category has now many copy-cater, that have more modern design and better default shortcuts, though that depends on the user.

Especially with agents, Multiplexer got its new category with working more inline with them, see the dedicated category below.

# Timeline

gantt
    title Terminal Multiplexers with first public release
    dateFormat YYYY-MM-DD
    axisFormat %Y

    section Classics
    GNU Screen      :milestone, 1987-01-01, 0d
    dtach           :milestone, 2004-08-01, 0d
    tmux            :milestone, 2007-11-20, 0d
    Terminator      :milestone, 2007-12-01, 0d
    dvtm            :milestone, 2007-12-07, 0d
    byobu           :milestone, 2009-01-01, 0d
    tmate           :milestone, 2014-01-01, 0d
    abduco          :milestone, 2014-07-05, 0d

    section Modern
    kitty Sessions           :milestone, 2017-02-02, 0d
    WezTerm         :milestone, 2019-03-24, 0d
    Zellij          :milestone, 2021-01-21, 0d

    section Agent-era
    workmux         :milestone, 2025-11-04, 0d
    cmux            :milestone, 2026-01-28, 0d
    Herdr           :milestone, 2026-03-27, 0d

# What is a Terminal Multiplexer?

A terminal multiplexer is a tool that lets you run multiple shell sessions, split your screen into separate panes, and keep your programs running in the background even if you lose your connection.

Three capabilities usually get bundled together, and each tool picks a different subset:

Session persistence: detach/reattach, survives SSH drops (screen, tmux, abduco, dtach)
Window/pane splitting: tiling inside one terminal (dvtm, tmux, Zellij)
Workspace/state layer: layouts, plugins, agent-state awareness (Zellij, Herdr)

# Tools

# Classics (server-side, SSH-safe)

GNU Screen: the actual OG, 1987. Preinstalled on nearly every Unix box, so it’s the fallback on locked-down legacy servers.
tmux: OG multiplexer (as far as modern usage goes). Huge plugin ecosystem via TPM, scriptable, works everywhere.
byobu: not a multiplexer itself — a config/status-bar wrapper over screen or tmux. Started life as screen-profiles in Ubuntu.
dtach: only the detach/reattach part, nothing else. ~1000 LOC.
dvtm + abduco: the suckless split — dvtm does dwm-style tiling in the console, abduco does session management. Together a simpler, cleaner alternative to tmux/screen.
tmate: tmux fork for instant terminal sharing over a relay. Pair-programming / remote support.

# Modern rewrites

Zellij: A terminal workspace with batteries included. Rust, discoverable keybindings shown in a status bar, WASM plugin system, works out of the box without a config file. The main tmux challenger.
(not out yet) Superlogical: Building the multiplexer for all work, company behind Mitchell Hashimoto

# Terminal emulators with built-in multiplexing

kitty Sessions, not as powerful, but might be enough. Shipped in kitty v0.1.0 (Feb 2017), a startup-layout file, not a persistence layer. Also has remote control for scripting panes.
WezTerm: GPU-accelerated emulator with a real multiplexer built in, including SSH domains. Needs a GUI though, so it doesn’t replace tmux on a headless box.
Ghostty and Mitchell Hashimoto want to build their own too.
Terminator: GTK emulator with tiling. Local-only, no session persistence — technically a tiling emulator, not a multiplexer.

# Agent-era

With Agents a new category emerged with more focus on shifts from just “split screens” seeing “What are my agents doing”?.

Herdr: The newest, more focused on Agents. The runtime coding agents run on. Rust, single binary, tmux-grade persistence + per-agent state detection (working / blocked / done / idle) + a scriptable socket API. AGPL-3.0 with a commercial option.
cmux: native macOS app built on libghostty — vertical tabs, notifications, restores agent sessions on relaunch, full CLI + Unix socket API including a programmable browser pane. macOS-only, no server-side SSH persistence.

# Configs that work with different Multiplexer

workmux: Turn your terminal into a multi-agent workspace. Every task gets a dedicated git worktree and tmux window, letting agents code conflict-free. Great video for more info.
TPM / oh-my-tmux: the standard tmux config starting points.

# Further Reads

Tiling Window Manager

Origin:
References: Terminals, My Terminal Workflow with MacOS and Linux

Data Engineering Acquisitions (2022-2026)

Mon, 31 Aug 2026 12:42:56 +0000

Consolidation in the Data Engineering market is happening quickly. Tools from the Modern Data Stack get unified into bigger Data Platforms. This note highlights the latest acquisitions across data engineering. It serves as an overview of the latest consolidations.

Find attached the acquisition overview from 2022 to today.

gantt
    title Data Engineering Acquisitions Timeline 2022-2026
    dateFormat YYYY-MM
    axisFormat %b %Y
    
    section Fivetran + dbt
    dbt → MetricFlow                     :milestone, 2023-02, 0d
    dbt → SDF                            :milestone, 2025-01, 0d
    Fivetran → Census                    :milestone, 2025-05, 0d
    Fivetran → SQLMesh                   :milestone, 2025-09, 0d
    Fivetran → dbt                       :milestone, 2025-10, 0d
    Fivetran → Great Expectations (Steward)         :milestone, 2026-05, 0d
    
    section Databricks 
    Okera                   :milestone, 2023-05, 0d
    MosaicML                :milestone, 2023-06, 0d
    Arcion                  :milestone, 2023-10, 0d
    Tabular                 :milestone, 2024-06, 0d
    Neon                    :milestone, 2025-05, 0d
    
    section Snowflake 
    Streamlit                :milestone, 2022-03, 0d
    Applica                  :milestone, 2022-08, 0d
    SnowConvert              :milestone, 2023-01, 0d
    LeapYear                 :milestone, 2023-02, 0d
    Neeva                    :milestone, 2023-05, 0d
    TruEra                   :milestone, 2024-05, 0d
    Datavolo                 :milestone, 2024-11, 0d
    Crunchy Data             :milestone, 2025-06, 0d
    Select Star              :milestone, 2025-11, 0d
    
    section IBM + Confluent 
    IBM → Confluent         :milestone, 2026-03, 0d
    Databand.ai       :milestone, 2022-06, 0d
    Immerok                  :milestone, 2023-01, 0d
    WarpStream               :milestone, 2024-09, 0d
	
	
    section Qlik 
    Talend                        :milestone, 2023-01, 0d
    Kyndi                         :milestone, 2024-01, 0d
    Upsolver                      :milestone, 2025-01, 0d
	
    section Analytics & BI
    Alteryx → Trifacta                   :milestone, 2022-01, 0d
    Hex → Hashboard                      :milestone, 2025-04, 0d
    Coalesce.io → CastorDocs             :milestone, 2025-03, 0d
    MotherDuck → Tower.dev             :milestone, 2026-08, 0d
    AWS → DuckLabs            :milestone, 2026-08, 0d
    
    section Data Quality & Observability
    Bigeye → Data Advantage Group        :milestone, 2023-06, 0d
    Datadog → Metaplane                  :milestone, 2025-04, 0d
    Soda → nannyML                       :milestone, 2025-06, 0d
    
    
    section Streaming & Real-time
    Cloudflare → Arroyo                  :milestone, 2025-04, 0d
    Redis → Decodable                    :milestone, 2025-09, 0d
    
    section Database & Infrastructure
    NetApp → Instaclustr                 :milestone, 2022-04, 0d
    Vector Capital → SingleStore         :milestone, 2025-09, 0d
	
    section SAP
    Dremio         :milestone, 2026-05, 0d

^4d105f

# Timeline

The timelines show each year’s acquisitions, starting in 2022. After that, we discuss related topics such as Bundling vs. Unbundling and the general state of the data engineering ecosystem.

# 2026

2026-08-26: AWS -> DuckLabs
- Announcement: AWS to acquire DuckLabs, the Amsterdam-based company behind DuckDB. We are not acquiring the DuckDB open source project, which will remain free and open source under the independent DuckDB Foundation (the non-profit that oversees DuckDB) and available under the MIT license as it does today. ^319c65
  - DuckLabs: DuckLabs – DuckLabs to Join AWS, Projects to Remain Open Source
    - What will change: “Joining AWS will give us greater capacity to invest in both the technology and the community around it. Going forward, the DuckDB Foundation will include a technical advisory board, so that leading community members can provide their input on the project’s technical direction. We also plan to open the extension stack so that extensions signed by other developers and organizations can run in DuckDB.”
  - AWS: AWS to acquire DuckLabs, the company behind DuckDB
2026-08-25 MotherDuck -> Tower.dev
- Announcement: MotherDuck acquires Tower.dev
2026-07-13: Prefect -> Dagster
- Announcement: Prefect acquires Dagster Labs, Nick Schrock ^04bb81
  - See more differences between Dagster and Prefect (and Airflow) with and FAQ about the merger
2026-05-08: SAP -> Dremio ^44670f
- Announcement: SAP to Acquire Dremio (The transaction is still pending regulatory approval.)
  - With Dremio, SAP will deliver a universal, Open Catalogs built on Apache Polaris and the open Apache Iceberg REST Catalog API. It serves as both the discovery and semantic layer of SAP Business Data Cloud, giving every connected engine – SAP or non-SAP – a single point of access to unified business context: meaning, relationships, access rights and data lineage. This catalog will form the foundation of the SAP Knowledge Graph, embedding business relationships, organizational hierarchies, regulatory classifications and cross-system lineage as native properties.
  - 2026-07-06: Complete: SAP Completes Dremio Acquisition
2026-05-07 Undisclosed Company -> Great Expectations (Fivetran didn’t acquire, but becomes steward of open source community and GX Core project)
- Annoucement: Great Expecations, Fivetran
  - GX Core will continue as an open source, community-driven project. Fivetran intends to support ongoing project maintenance, ecosystem integrations, and community engagement.
2026-03-17: IBM -> Confluent^e4f448
- Announcement: 2026-03-17: Finish Acquisition, 2025-12-08: IBM start to aquire
2026-03-19: Astral (The company behind uv, ruff, ty) -> OpenAI
- Announcement: Astral: Astral to join OpenAI, OpenAI: OpenAI to acquire Astral | OpenAI

# 2025

2025-12-16: Matatika -> Meltano (Arch) ^61399b
- Announcement: LinkedIn Post
2025-11-24: Snowflake → Select Star
- Announcement: Select Star Joins Snowflake: A New Chapter for AI + Data Discovery and Snowflake to Acquire the Select Star Technology to Expand Horizon Catalog’s View of Enterprise Data for Next-Gen AI
2025-11-18: DVC → LakeFS ^c38a54
- Announcement: A Celebration of Shared Vision: lakeFS + DVC
2025-11-04: ClickHouse → LibreChat
- Announcement: ClickHouse welcomes LibreChat
2025-10-13: Fivetran → dbt ^746334
- Announcement: dbt Labs + Fivetran: Open data Infrastructure for analytics and AI | dbt Labs
  - latest updates at dbt
2025-09-10: Vector Capital Management → SingleStore ^b9f630
- Announcement: SingleStore Announces Growth Buyout Led by Vector Capital
2025-09-04: Redis → Decodable ^413719
- Announcement: Redis to acquire Decodable
2025-09-03: Fivetran → SQLMesh
- Announcement: Acquisition (SQLMesh)
2025-06-13: Soda → nannyML
- Announcement: Announcement
2025-06-02: Snowflake → Crunchy Data ^193bc3
- Announcement: Announcement
- Deal Value: Estimated the deal to be worth around $250 million. Delivering the Most Enterprise-Ready Postgres, Built for Snowflake
2025-05-14: Databricks → Neon ^3cf8ff
- Announcement: Announcement
- Deal Value: Bought Neon for a reported $1 billion in equity ( source)
2025-05-01: Fivetran → Census
- Announcement: Fivetran Signs Agreement to Acquire Census
- Blog: Why Fivetran and Census are joining forces
2025-04-30: Hex → Hashboard
- Announcement: Hex Acquires Hashboard
- Blog: Hex has acquired Hashboard
2025-04-23: Datadog → Metaplane
- Announcement: Datadog acquires Metaplane
- Blog: Metaplane by Datadog
2025-04-10: Arroyo → Cloudflare
- Announcement: Arroyo is joining Cloudflare | Arroyo blog
- Comment: To build R2 SQL
2025-03-19: Coalesce.io → CastorDocs
- Announcement: Coalesce Expands Data Platform With CastorDoc Acquisition
- Blog: Building the Future of Data Together: Why Coalesce Acquired CastorDoc
2025-01-14: Qlik → Upsolver
- Announcement: Qlik Acquires Upsolver
2025-01-14: dbt Labs → SDF
- Announcement: dbt Labs Acquires SDF Labs
- Blog: dbt Labs acquires SDF Labs to advance analytics engineering

# 2024

2024-11-20: Snowflake → Datavolo
- Announcement: Snowflake Agrees to Acquire Open Data Integration Platform Datavolo
- Blog: Snowflake snaps up data management company Datavolo
- Deal Value: Approximately $110M (primarily stock, remainder cash)
- Comment: Powered by Apache NiFi, provides single platform for automating and managing both structured and unstructured data flows
2024-09-09: Confluent → WarpStream
- Announcement: Confluent Acquires WarpStream to Advance Next-Gen BYOC Data Streaming
- Blog: Confluent Acquires WarpStream
- Deal Value: $220M
2024-06-04: Databricks → Tabular (Iceberg) ^2f6984
- Announcement: Databricks Agrees to Acquire Tabular
- Blog: Databricks + Tabular ^6a38ee
2024-05-22: Snowflake → TruEra
- Announcement: Snowflake Acquires TruEra to Bring LLM & ML Observability to Data Cloud
- Blog: Snowflake acquires TruEra to deliver LLM observability inside Data Cloud
- Comment: Provides tools to test, debug, and monitor ML models and LLM apps in production
2024-03-19: Databricks → Lilac AI
- Announcement: Lilac Joins Databricks to Simplify Unstructured Data Evaluation for Generative AI
- Blog: Databricks acquires Lilac to supercharge data quality efforts for gen AI apps
2024-01-30: Databricks → Einblick
- Announcement: Welcome to the Data Intelligence Platform, Databricks + Einblick
- Blog: Databricks acquires team behind natural language data notebook Einblick
2024-01-18: Qlik → Kyndi
- Announcement: Qlik Acquires Kyndi to Advance AI-Driven Business Outcomes

# 2023

2023-10-23: Databricks → Arcion
- Announcement: After 43B valuation, Databricks acquires data replication startup Arcion for 100M
- Deal Value: $100M
- Comment: Data replication
2023-06: Databricks → MosaicML
- Announcement: Databricks Completes Acquisition of MosaicML
- Deal Value: $1.3B
- Comment: One of the biggest AI deals of 2023
2023-06-22: Bigeye → Data Advantage Group
- Announcement: Data observability platform Bigeye announces acquisition of Data Advantage Group
2023-05-24: Snowflake → Neeva
- Announcement: Snowflake Acquires Neeva to Accelerate Search in the Data Cloud Through Generative AI
- Blog: Snowflake acquires Neeva to bring intelligent search to its cloud data management solution
- Deal Value: Approximately $150-185M
- Comment: Founded by former Google executives; founder Sridhar Ramaswamy became Snowflake CEO in February 2024
2023-05-03: Databricks → Okera
- Announcement: Welcome, Okera: Adopting an AI-Centric Approach to Governance
- Blog: A new chapter in our journey: Okera is joining the Databricks family
- Comment: Okera co-founder and CEO Nong Li, creator of Apache Parquet, joined Databricks
2023-02-07: Snowflake → LeapYear Technologies
- Announcement: Snowflake to Acquire LeapYear
- Deal Value: $62M
- Comment: Differential privacy platform to enhance data clean room capabilities
2023-02-08: dbt → MetricFlow (Transform.co)
- Announcement: dbt Labs Signs Definitive Agreement to Acquire Transform, Accelerating development of the dbt Semantic Layer (updated by dbt)
- Press: Dbt Labs acquires Transform, adding semantic tools to its data analytics platform
- GitHub MetricFlow: The Future of the MetricFlow Project
2023-01-19: Snowflake → SnowConvert (from Mobilize.Net)
- Announcement: Snowflake Announces Intent to Acquire Mobilize.Net’s SnowConvert
- Deal Value: 76.3M (approximately 11.6M net of cash acquired)
2023-01-06: Confluent → Immerok
- Announcement: Confluent Announces Intent to Acquire Immerok to Accelerate Development of Cloud Native Apache Flink
- Deal Value: Approximately $54.9M
- Comment: Founded by original creators of Apache Flink
2023-01-05: Qlik → Talend
- Announcement: Qlik Acquires Talend
- Comment: Both companies owned by Thoma Bravo; Talend was acquired by Thoma Bravo for $2.4B in 2021

# 2022

2022-08: Snowflake → Applica
- Blog: Snowflake blog
- Comment: Specializes in decoding and automating complex documents based on deep learning
2022-06-27: IBM → Databand.ai
- Announcement: IBM Aims to Capture Growing Market Opportunity for Data Observability with Databand.ai Acquisition
- Deal Value: Estimated ~$150M
- Comment: Data observability platform; validated data observability as critical category in modern data stack
2022-04-07: NetApp → Instaclustr
- Announcement: NetApp acquires Instaclustr to deliver open source databases as a service
- Comment: Managed open source databases as a service (PostgreSQL, Kafka, OpenSearch, Cassandra)
2022-04: Talend → Gamma Soft
- Comment: Cloud-based data discovery platform based in France, founded 1983
2022-03-02: Snowflake → Streamlit
- Announcement: Snowflake acquires Streamlit for $800M
- Deal Value: $800M
- Comment: Popular open source project for building data-based applications
2022-01-06: Alteryx → Trifacta
- Announcement: Alteryx Announces Acquisition of Trifacta
- Deal Value: $400M cash + 75M retention pool (RSUs)
- Comment: Cloud-based data preparation platform

# Bundling vs. Unbundling

Bundling vs Unbundling

# Once Called, Software is Eating the World

Once called “Software is eating the world”, now it seems the pendulum is swinging back to more unified and integrated data platforms.

I think the best way for OSS products to survive is to embrace the «Declarative Data Stack» approach, where integration happens with a single configuration file. By integrating with multiple tools, you gain the best of both worlds: a combination of integrated and open-source capabilities, along with end-to-end analytics.

However, making money from open-source is hard, but I still hope that many will pursue this path. When deciding on a tool, I will always pick the open-source one. To me, it builds trust, and because it’s shared as a gift for free to use by anyone, it makes me want to support it more.

Consider the Framework Laptops. They’re fully repairable, with every part replaceable, allowing you to swap out the screen or even the motherboard later on. This is something I want to support. Same with OSS. I believe the strategy shouldn’t be to cash out on OSS, but rather have it as a sign of valuing your customer by giving them a gift.

The money stream should be independent of OSS, if at all possible, so you have a clear distinction, and also don’t confuse them with the company down the road. Much easier said than done, but there are still many great data engineering companies that are doing a great job of exactly that. I hope it stays this way.

What do you think, should the vendor still have an open-source product, or focus on making money so it will sustain itself over time?

# Fivetran + dbt

Fivetran + dbt Merger

# Table Formats Market Updates

See Open Table Formats (Market Updates).

# AI Acquisition

Seperate note worthy acquisition in AI related to data engineering.

2026-08-27: NVIDIA -> HuggingFace ^117aa1
- Announcement: Nvidia agrees to buy Hugging Face for $12.9 billion, The Information reports
2026-04-22: SpaceX -> Cursor AI IDE
- X Announcement: > Cursor has also given SpaceX the right to acquire Cursor later this year for $60 billion or pay $10 billion for our work together. NY Times
- Update 2026-06-16: SpaceX locks in $60 billion Cursor deal to close gap with rivals in AI coding race | Reuters
- Update: 2026-08-14 it’s finished: Cursor Tweet
2026-05-06: Codeium, the makers of the Windsurf AI IDE, forked from VSCode were acquired by OpenAI for $3 billion dollars

# Further Reads

Origin: Data Engineering
References: Earning Money with the Open-Source Model—Making Gifts

Fonts: Reading/Writing and Programming Monos

Mon, 31 Aug 2026 08:46:39 +0000

I have a love for nice fonts.

# Reading and Writing Fonts

Atkinson Hyperlegible (mentioned Your favourite fonts choice and why? thread - Share & showcase - Obsidian Forum )
iA Writer Quattro
Inter - Default of Obsidian and used also at Airbyte website
Fonts - DYSLEXIA which makes reading easier for people with problems (from Tobias Ziegler (Schäuble))
Piazzolla, serif; from How to treat monospace alike fonts in a tech blog? - Pimp my Type
- Piazzolla Type System
Gentium Book Basic, serif: Discovered on Why Write?, beautiful. (Free for personal use and free for commercial use)
Also beautiful: SN Pro ( origin)
UxumGrotesque, sans-serif ( origin)

Atkinson Hyperlegible vs Piazzolla:

Updated fonts in Obsidian:

# Programming Fonts

Here are some tips for programming (from RW 15 Best Programming Fonts for Better Coding):

Clear and easy-to-read characters to reduce eye strain when spending hours looking at hundreds of lines of code.
Makes a clear distinction between commonly-confused characters such as the letter “O” and the number “0” or the lowercase “L” and the number “1”.
Ligatures or extra whitespace for commonly-used symbols in popular programming languages – not for everyone, but others love it.
Fonts with multiple variations on how certain characters are handled are great, so you can pick and choose exactly the version you prefer.

Testing out fonts: Programming Fonts - Test Drive.

# Open Source Favorites

Iosevka - My current one, 14.7k start on GitHub 😳
- The two quasi-proportional families: Iosevka Aile, and Iosevka Etoile, are made for documents and writing. They provide more variety on character widths, including a slightly wider letter w and m, and narrower i and l.
- to check out, Berkeley + iosevka = IoskeleyMono
Source Code Pro for Powerline - Adobe open source
Cascadia Code - default font for the Windows Terminal and Visual Studio
Anonymous Pro for Powerline - I like
- Four styles: Regular, Italic, Bold, and Bold Italic.
- Fixed width design.
- Various computer keyboard symbols.
- DOS box-drawing characters.
- Extended language support for most Latin-based Western and Central European languages, as well as Greek and Cyrillic.
Zed Mono looks nice too: GitHub - zed-industries/zed-fonts: The Zed Mono and Sans typefaces, custom built from Iosevka Origin

Some comparisons while testing:

A nice summary that has similar preferences that are similar to mine:

Some of my favourite monospace / coding fonts Link

Cascadia Code 81 ← What I use in my terminal

JetBrains Mono 63 ← What I use in VS Code and other editors

Iosevka 128 ← Fantastically comprehensive and space efficient

Anonymous Pro 57 ← A classic, but no ligatures

# Payed

Berkeley Mono (origin: This Tweet)
Gintronic so far one of my favorite

# Closed

Monaco looks beautiful see Reddit

# Customize

A webapp to customize certain parts of the fonts: Iosevka Customizer.

# Create Font from Handwriting

FontCrafter: Create Your Handwriting Font for Free

# Finding Your Coding Font

codingfont.com
- my outcome:
  - but Iosevka, my favorite, was not part of it! :)

# Integrate into Website

# Others

DM Mono and Roboto Slab

# Further Reads

Text Font Generators

Origin:
References: Writing, New Tabs in Obsidian with version 1.0

Open Source (OSS): Advantages, Challenges, and AI

Mon, 31 Aug 2026 08:44:38 +0000

Open source software refers to software projects shared online on GitHub, Codeberg, Tangled, or other GitHub Alternatives, usually with a git repository. Similarly, modifications and derived works are allowed based on Licensing Models.

The acronym of OSS

In our context it refers to Open Source Software. Wikipedia

# The Hook of Open Source

I’ve been a big fan of open source since very early on, when I worked in Business Intelligence, and all I had to use was closed-source software, and you could only try it after talking to a sales agent. With open-source, this was different. The code wasn’t perfect or running smoothly, but I could try it, get a feel for it, fork it, change it, and expand on it. That’s what really hooked me early on.

Today (2026-08-18), two decades later, everything has changed; see some more thought on How is it Changing with AI below, but I still think OSS is what is best for humanity in general. Why build 100 of the same things individually? Why not work together and make one great product?

# Why Open Source?

Given that a single organization can’t create and maintain every data connector, framework, or component along the data stack, open source is a method to tackle the challenges in a new way together.

Let’s look at a few software companies as examples. Red Hat, Elastic, and Cloudera have implemented successful open-source strategies. Airbnb has more than 30 open-source projects, and Google has more than 200. In data, Databricks open-sourced Apache Spark, so now everyone has access to its internals.

The beauty of open-source is that the more you share, the more feedback, contributions, and, most importantly, trust a product receives. Usually, open-sourcing improves a product, as community contributions can’t compare to those from a small team of in-house engineers.

Even though it may seem counterproductive to share code publicly, as competitors may try to replicate it, doing so may be the most effective way to foster collaboration and creativity. The pressure to turn a profit is ever-present. Still, there is ample opportunity to build AI-powered, business-driven innovations on top of a data stack that strengthens with each new open-source contribution.

# What’s the Advantage of Open Source?

The advantage of OSS is not really that it’s “free” (that is relative in IT), but strategic with interoperability — says Matthias Stürmer in 1 zu 1 mit Matthias Stürmer, BFH - 1 zu 1 - der Bedag Fachtalk | Podcast on Spotify (more).

# Challenges of Open Source

One of the biggest challenges, and a common one, is “how to make money”? We saw software is eating the world with Why Software Is Eating the World by a16z, to open-source strategies in Silicon Valley VC strategies, to give everything for free, build a great product, and figure out the finances later. But that’s not as easy.

Sponsoring would probably be best, or if the repository is big enough, e.g., the one that started it all, the Linux repo, you get status and might get another job or so. But it’s still, to this day, not a solved problem.

# How OSS Makes Money

I like how 37Signals handles it, they produce 100 of repos with Kamal, Fizzy, Omarchy, Shape Up and so much more. But I heard them saying that first you make the money, and only then you share it. And after that, you don’t expect any return. DHH called it a Make Gifts, this is all open source is. Because others can take it or leave it, but can’t demand that you do things. It’s a gift to the world.

I like this framing, too.

What open source also enforces is a focus on Openness and Open Standards. If you become the standard through openness and transparency, you might capitalize down the road with services or just good marketing.

# The Opposite of OSS: Proprietary Software

Technically, “closed-source” is not the right opposite. Proprietary software is more correct, and it’s about licenses, which define what can be used and how.

# Why Open Source Issues Should Be Created with Real Names

Good write-up on Dear anonymous internet user asking for help.. - Bert Hubert’s writings:

First, I just consider it rude. You come at me hiding who you are but still expect me to do free work for you. Try doing that in real life. What were you thinking? Not introducing yourself AND using a fake identity?

Found it linked in the DuckDB issue creation.

# The Fun and Joy of OSS

The joy of contributing to an OSS project. Building something for yourself, but sharing it and getting it distributed to 1000s. Not because of anything, but for the fun and joy of it. Very rewarding. That’s why I believe the OSS-based strategy will never die, despite the dbt/Fivetran merger’s efforts to convince us otherwise. My Bsky post ^1cb4bd

# How is it Changing with AI

AI is the best and worst thing that has happened to open-source at the same time. As a user, you can customize every piece of software by just prompting the cloned git repo. E.g., Omarchy is my OS, and I can just prompt to change everything in my OS; that is amazing.

The worst part is from the maintainer’s perspective—so many PRs and noise to deal with. Also, programming is leading to AI exhaustion, as you typically work more and just less satisfied. You don’t even know what you’ve built anymore.

A great video highlighting all of it is here by ForrestKnight.

# A Great Example: Agentic OS

With Omarchy as the first Agentic OS, you basically prompt your Agent to add a widget for time zones, or a stats panel for your website, or a weather radar warning with satellite images, directly built into your OS. All of these are plugins I’ve built or installed from the Omarchy Marketplace that people quickly prompted.

And the key here is Open Source. Because Linux, the foundation of Omarchy is open source, it’s in the training data of every model, the Omarchy provides skills for how to build plugins with clear instructions to use Quickshell, a Flexible toolkit for making desktop shells that uses something called QtQuick that lets you build easily with C++. Because agents know that language well, it’s super easy and super fast, with C++ being one of the fastest Programming Languages out there.

# COVID: Collaborativeness of the Whole World

Another good example was COVID-19, where a vaccine was developed faster than ever before, with people working together all over the world. Not really open source in the sense of code, but sharing the learnings openly among labs is the same spirit.

When resources are limited, we need to work together and share our findings instead of re-creating the same at every company again and again.

# Further Reads

How open-source envolved explained in Open Source: From Community to Commercialization by a16z (internal)
Open source security at Astral (company behind uv)
Supporting Open Source
Open-Source Data Engineering Projects

Origin:
References: Open Source at Airbyte

Declarative Data Stack

Mon, 31 Aug 2026 08:35:05 +0000

Declarative Data Stack is a term introduced in the article The Rise of the Declarative Data Stack by Mike Driscoll and myself.

Below you find more about what it means, and what declarative data stacks are out there already.

# What Is a Declarative Data Stack?

A declarative data stack is a set of tools and, precisely, its configs can be thought of as a single function such as run_stack(serve(transform(ingest))) that can recreate the entire data stack.

Instead of having one framework for one piece, we want a combination of multiple tools combined into a single declarative data stack. Like the Modern Data Stack, but integrated the way Kubernetes integrates all infrastructure into a single deployment, like YAML.

We focus on the end-to-end Data Engineering Lifecycle, from ingestion to visualization. But what does the combination with declarative mean? Think of Functional Data Engineering, which leaves us in a place of confident reproducibility with little side effects (hopefully none) and uses idempotency to restart function to recover and reinstate a particular state with conviction or rollback to a specific version.

More on The Rise of the Declarative Data Stack.

Other Naming

Tobiko calls it an Integrated Data Stack.

Dagster talks about Impedance Mismatch, and Data Asset oriented orchestration: The Data Engineering Impedance Mismatch | Dagster Blog

# Why you need a Declarative Data Stack?

Why do you need an declarative data stack or Open Enterprise Data Platform, you might ask?

Data grows yearly more than the entire lifetime before. There is a growing need to make sense of more data. In the old days, you had a single vendor solution: think of SAP or Oracle. These days, new SaaS and open-source tools products created daily, specializing in a tiny little niche. So why would you need another platform?

You want the best of both worlds. You want Open Source not to be locked in and to use the strongest, collaboratively created tools in the open. People worldwide can fix a security bug in minutes or fix it yourself—compared to an extensive vendor where you solely rely on their update cycle.

The downside of open-source is that there are a lot of bugs, missing features, and independent tools. That’s precisely where HelloDATA BE comes into play. We are building the missingplatform that combines the best-of-breed open-source technologies into a single portal, making it enterprise-ready.

# How to balance flexibility with the simplicity

Flexibility is one disadvantage of DDS as you may need to implement the interface first to respect new functions in your configs. Therefore, if you require full flexibility, an imperative approach is preferable.

I’d say start somewhere, try it out, feel it, and adjust. If you know you’re going to throw it away, there’s no need for a DDS. However, if you know you are at the enterprise level, you most probably want the simplicity and also the restrictive nature of DDS. asked on LI

# Tools

# Close-Sourced DDSEs

Let’s start with closed-source first—one key point to note. Most of what we’ve discussed here is something that most closed-source tools have implemented in one way or another. Because they’ve built one big monolith, this is relatively straightforward and the natural thing to do.

This can be more challenging and not immediately obvious with an open-source approach and numerous integration tools. Let’s now look at tools that have successfully implemented such features.

Ascend: The platform automates up to 90% of repetitive data tasks using their DataAware Automation Engine.
Palantir Foundry: One of the first lakehouse implementations before the term was coined. Enables real-time collaboration between data, analytics, and operational teams through a common logic data lake layer
Find more on Closed-Source Data Platforms and a fantastic read on composable data stacks on a new frontier by Voltron Data.
Y42: Mission Control for Your Data Pipelines.

Usually, the problem with closed-source software is that it is structured as a monolith, combining transformation logic with persisted database tables while keeping the underlying code unknown.

# Open-Source DDSEs

But even more interesting are the open-source tools I found[^1] - they are fantastic and built in the open, building in the open. Not all might be truly declarative data stacks by their definition, but they all build on top of other tools and declaratively integrate them.

DataForge: Write functional transformation pipelines by leveraging software engineering principles. It does not have a visualization tool that focuses on transformation[^3].
Starlake: Declarative text based tool for data analysts and engineers to extract, load, transform and orchestrate their data pipelines.
Dashtool: A Lakehouse build tool that builds Iceberg tables from declarative SQL statements and generates Kubernetes workflows to keep these tables up-to-date. It handles Ingestion, Transformation, and Orchestration. Written in Rust and uses Datafusion.
BoilingData: A local-first data processing native application designed for rapid data pipeline development. Enables data engineers to build and test pipelines quickly using tools like DuckDB, dbt, and dlt.
HelloDATA BE: An enterprise data platform built on open-source tools based on the modern data stack. It uses state-of-the-art tools such as dbt for data modeling with SQL and Airflow to run and orchestrate tasks, Superset to visualize the BI dashboards, and JupyterHub for data science tasks. It includes multi-tenancy, full authentication, and authorization, which are handled with a single web portal.
GitHub Actions: This is the simplest version for building a declarative data stack. A deploy.yaml script could be a simple DDS config. GitHub also has an engine that converts and runs on Docker-runners. So, in a way, it’s another engine implementation, and maybe we could take some configs based on that? (see image at github-actions-dds.webp)
Datacoves: The platform helps enterprises solve data analytics challenges with managed dbt, Airflow, and VS Code, adopting best practices. This approach avoids negotiating multiple SaaS contracts and reduces consulting costs without compromising data security.
Datadex: Serverless and local-first Open Data Platform.
BigFunctions: A framework to build a governed catalog of powerful BigQuery functions, SQL first approach. Ingesting, advanced data transforms, and serving data on a data app with a single SQL query.
Hoptimator: Multi-hop declarative data pipelines.
Bruin: Build data pipelines with SQL and Python, ingest data from different sources, add quality checks, and build end-to-end flows.
Iasql:: Cloud Infrastructure as data in PostgreSQL.
SQL Declarative-Data-Stack such as BigFunctions.
Exasol Personal: The Analytics Database for Agentic AI. Free for Personal Use. Starterkit

# Templates

These are not actual declarative stack, but templates to deploy a data stack.

Boring Data: An AWS + Snowflake Data Stack Template by Julien Hurault.
- Documentation at docs.boringdata
Nexus-Stack: One-command deployment: Hetzner + Cloudflare Zero Trust + Docker by Stefan Koch. Acompanied blog post at Nexus Stack: Your Data. Your Rules. Your Flow.

Existing Templating

Beyond tools, templating can solve some of the Jinja Template, GoLang’s template package, biGENIUS, Data Warehouse Automations (DWA), Template modules, Apache Velocity, Liquid, and many others.

# Compilers and Frameworks

SDF: Similar to DataForge, built on Rust and Datafusion. Tries to be the Typescript for SQL, creating faster development cycles and reliable results with a powerful compiler.
SQLMesh: An efficient data transformation and modeling framework that has compiler capabilities built-in with SQLGlot, a Python SQL parser and a transpiler.

SQL Compilers

SQLGlot would be a good integration to parse SQL without running. Same as SDF integration with Datafusion.

# Lists

GitHub List with open-source declarative data stacks at sspaeti’s list - Declarative Data Stacks.

# Standard

Open Standards are important.

Something like Open Transformation Specification (OTS) would help for tranformations (ETL) amoung above tools who created already standards in their domains. Or Open Semantic Interchange (OSI) for semantics, etc.

# Alternatives

# What’s the difference between a Composable Data System

Composable Data Stacks or System sounds very similar. Or also Multi-Engine Data Stacks. Are all of the same, but different wording?

# Testing

We need Deterministic Simulation Testing for DDS, like libSQL does for the sqlite rewrite.

# Declarative Data Stack ENGINE

Engine is an important part, which I go into more details in Designing a Declarative Data Stack: From Theory to Practice | ssp.sh, similar to Markdown can be the code, and HackMD, GDocs and other are engines to run it.

Key distinctions: part-3-example-implementation-declarative-data-stack

Docker and Kubernetes are other engines. There are many more, see part-3-example-implementation-declarative-data-stack.

# Further Reading

My Series
- The Rise of the Declarative Data Stack
- Designing a Declarative Data Stack: From Theory to Practice
Declarative data stack with Starlake: Configure, Don’t Code: How Declarative Data Stacks Enable Enterprise Scale
Related: Open Data Platform with Open Standards over Silos
From Data Engineer to YAML Engineer
with Modal: Building a cost-effective analytics stack with Modal, dlt, and dbt | Modal Blog
ETH in Zurich wrote a paper on “End-to-End Declarative Data Analytics: Co-designing Engines, Interfaces, and Cloud Infrastructure”, see Data Engineering Whitepapers. Tackling the the disconnect between Query Engines and cloud infrastructure.

Origin: Rill | The Rise of the Declarative Data Stack, The Rise of the Declarative Data Stack - Rill
References: Declarative Data Pipeline

VertiPaq (xVelocity): The In-Memory, Columnar Query Engine Powering Microsoft Analytics

Sun, 30 Aug 2026 06:47:30 +0000

The VertiPaq engine is an in-memory, columnar Query Engine that powers the data model in MS Excel Power Pivot, SQL Server Analysis Services Tabular, and Power BI. When data is loaded into a data model, the VertiPaq engine compresses and stores it in RAM for efficient processing.

Why isn’t there more information out there about VertiPaq Engine?

It’s interesting how little is public about how this engine works, yet it powers most of Microsoft BI and analytics solutions. This note includes background information gathered from social media over time. If you have some additional information, please reach out or join the conversation on Bluesky below.

# How Does it Work?

VertiPaq is a memory-optimized ColumnStore index; it is the same technology implemented in the SQL Server engine, in the form of non-clustered columnar indexes Source

Source: Inside the VertiPaq Engine by Marco Russo

This technology is comparable to the Photon Engine used by Databricks for their compute operations, as well as similar solutions employed by Snowflake and other platforms.

Another thing to note is that VertiPaq handles joins through pre-computed hash maps (called relationships) that are rebuilt anytime a table in the join changes. The hashmap sits in the CPU cache, and this avoids regular join performance concerns. Another read-focused design choice. This doesn’t apply for CCI in SQL Server or Synapse, just the read-heavy analytics-focused implementations of VertiPaq. Alex Barbeau on Bsky

# Different Versions

Got multiple flavors of xVelocity:

VertiPaq (xVelocity): Originally introduced as VertiPaq in 2006, this is the in-memory analytics engine that was integrated into Microsoft’s suite of data tools like Excel, PowerPivot, SQL Server Analysis Services (SSAS), and Power BI. It’s known for its columnar storage, which significantly speeds up data retrieval and analysis by optimizing data compression and query performance. Citation: VertiPaq vs ColumnStore Comparison - SQLBI
Optimized ColumnStore Index: This is another implementation of the xVelocity technology within SQL Server. Here, it’s used to create non-clustered columnar indexes, which are particularly effective for large data warehousing operations. The performance comparison between VertiPaq and ColumnStore indicates different scenarios where one might be preferred over the other for query performance. Citation: VertiPaq vs ColumnStore Comparison - SQLBI
V-Order with Microsoft Fabric: V-Order is a specific optimization applied to Parquet files within Microsoft Fabric, which enhances read performance under various compute engines like Power BI, SQL, and Spark. It’s not a direct version of VertiPaq, but it leverages similar principles of columnar storage and in-memory-like access for improved efficiency. V-Order applies sorting, row group distribution, dictionary encoding, and compression, which aligns with VertiPaq’s core philosophy of optimizing data for fast retrieval. Citation: Understand V-Order - Microsoft Fabric | Microsoft Learn
- Note: While V-Order is specific to Fabric and not directly a version of VertiPaq, it represents the ongoing evolution of Microsoft’s approach to data storage and retrieval, leveraging columnar structures for performance gains, much like VertiPaq initially did.

The 8-byte rule applies to all the following technologies: xVelocity memory-optimized columnstore indexes can see a 400% performance gain when using a data type of 8 bytes or less. Some historical knowledge in:

# Used in

More shared by Rif Kiamil:

Azure SQL Database: Supports columnstore indexes, which utilize xVelocity (VertiPaq) technology.
Azure SQL Managed Instance: Supports columnstore indexes, leveraging xVelocity technology.
SQL Server 2012 (On-Premises) and later: Introduced non-clustered columnstore indexes in SQL Server 2012, utilizing xVelocity technology.
Azure Synapse Dedicated SQL Pool (formerly SQL DW): non-clustered columnstore indexes. Azure Synapse’s Dedicated SQL Pool is tailored for its massively parallel processing (MPP Database) architecture,

# History

Based on the discussion in RW The rise and history of Power BI with Amir Netz by Kasper On BI between Amir Netz and Kasper On:

The VertiPaq development journey began around 2008 when Microsoft’s team sought to create a column store engine capable of delivering unprecedented performance. The breakthrough occurred when Amir Netz, inspired by an article stating that “data is not distributed uniformly in systems,” developed a revolutionary algorithm during one sleepless night. Christian Petculescu, one of Microsoft’s top developers, implemented the initial version, which comprised approximately 2 million lines of code.

VertiPaq’s revolutionary nature stemmed from its fundamental departure from traditional OLAP approaches. Rather than relying on disk-stored pre-computed aggregations, it used in-memory processing with innovative compression algorithms that could operate directly on compressed data. The engine was optimized for small to medium-sized datasets that fit in memory. Its initial performance tests were so remarkable that they raised skepticism, with Amir noting it performed “an order of magnitude faster than any other column store” – a distinction that remains true over a decade later.

The technology transformed Self-Service BI by eliminating the lengthy pre-computation times typical of traditional OLAP systems. This advancement let business users make changes and see results instantly, similar to the Excel experience. Amir considers VertiPaq a “once-in-a-lifetime” technological breakthrough, describing it as “perfect” – so well-optimized that its core algorithm has remained largely unchanged for over a decade. The development team carefully introduced it to avoid disrupting users’ existing technology adoption. This shift from disk-based, pre-computed OLAP systems to interactive, in-memory analytics proved crucial for Power BI’s success and modern business intelligence.

# Renaming

From 2022-03-19: VertiPaq has been rebranded to xVelocity. The official name is now “xVelocity in-memory analytics engine (VertiPaq)”, though both terms – xVelocity and VertiPaq – are used interchangeably when discussing Analysis Services. Source

# 2026-08-30

Alex Dupler on X:

It’s still being developed. There are entire books about it. But the public names are Power BI Semantic Models and Analysis Services Tabular. Vertipaq is technically the internal name for the storage engine component in its primary mode (imported data).

Recent major enhancements include added user defined functions to the query language, and “direct lake mode” where delta tables can be transpiled to vertipaqs in memory metadata with out recompression, for zero copy near real time querying of Delta tables.

# Other In-Memory Formats/Engines

Shared on Twitter

See more on In-Memory Formats.

Apache Parquet Reader: Writing Parquet That VertiPaq Likes – Small Data And self service

# Further Readings

Jeffrey Wang is heavily involved and writes on pbidax with articles such as Understanding the Impact of Value Filter Behavior – pbidax
People involved: Amir Netz, Marco Russo and Alberto Ferrari
- Costs of Relationships in DAX - SQLBI
The VertiPaq Engine in DAX | Microsoft Press Store

Origin: Tweet
References: In-Memory Formats

Learn for Life

Fri, 28 Aug 2026 10:56:21 +0000

Learn for life was my earliest Mantra and slogan in my blog. It has stayed with me to this day; it’s My Principles number 1. Anthony de Mello says in Awareness (Anthony de Mello, J. Francis Stroud), that this is ke to waking up.

It was for a long time on my blog, and as a background image on social media:

Jason Fried said ^99a7ca

“That’s where they (the insights) come from, the shower and the software”.

Having insights is the most fun—my highest principle is to learn something new every day. It’s the same Mantra as having an insight. We can push them by doing something new or in an unconventional way. Doing the Pathless Path, or going against the hype. Loved that conversation. ^695a9b

Alternative, better-flowing than Learn for Life

Always learning. (?)

# Defining Success

Common definition of success: reaching a desired outcome.
Scientist’s definition of success: learning something new. If you learn something, you haven’t failed.
- She explained how scientists predefine how long they experiment, not stopping if they don’t feel like it. E.g., if you say you want to write more, instead of just writing, you define: I will write for at least one month, or at least 10 articles. And then you don’t stop before.
- Success is not defined by the outcome, but by the experience, the way that you learned something new!

Snipped from How I Think - Anne-Laure Le Cunff - How to Design Tiny Experiments Like a Scientist - see my snippets.

# Further Reads

New ideas daily

Origin: Learn for Life in Bali.

Weekly Todo Lists

Thu, 27 Aug 2026 13:48:18 +0000

Similar to what Kepano said, The only to-dos method that has worked for me is weekly lists:

Every week I make a weekly note, and write my to-dos for the week. I may add more items to it during the week.

If any items didn’t get done I roll them over to the next weekly note or drop them.

On Monday, or sometimes on Sunday before the week, I write down all my upcoming tasks and things to take care of in the upcoming week on Physical Notebooks. I usually have a checkbox list in my physical notebook and divide it into an initial list of important tasks, with the most important one on top. Then divide it by the admin list below, which includes less important administrative tasks to send out invoices or handle.

I liked separating them so I can see my main work tasks in one list and assess how much they take me.

Also, sometimes it’s hard to start (see Ultradian Rhythm), so I just take a short admin task to start, that doesn’t require much brain power, and then it’s easier to me to get into the flow and feel “wow I already have finished two tasks” and continue to tackle the bigger once above.

I also have My Todos list digitally. The difference here is that I add tasks that I remember during work to quickly note down (I have a shortcut with space + space in Obsidian and Neovim to access this exact note quickly - see Open-Specific Files with Obsidian and Neovim) a task or important thing that came to mind.

This is more of an addition, and usually these tasks are much smaller and less important. The important tasks are on paper.

# Further Reads

Todos Need Constraints

Origin: Todos
References: How I write Daily Notes differently

Todo's: Need Constraints

Thu, 27 Aug 2026 13:47:50 +0000

One thing I learned about todos, it’s never about the todo app, it’s about having constraints to not add anything immediately. Important things will resurface again. Urgent matters in the moment, but irrelevant tomorrow, will save us a lot of time.

Tiago Forte or Getting Things Done (GTD) are doing a triage to enter everything, and then review them and only add what’s important or needed. But that never really worked for me. Now I have even more work. Besides I gave myself lots of todos, that seemed important during that time, I also need to go back and review them? Instead, I use much more often my Second Brain to dump ideas, to store thoughts, and only really add the real todos, and things I grappled with a lot.

E.g. don’t add shop for tomorrow, that’s not a todo. You need to shop, that’s a necessity. Also don’t add things you’d like to learn or wish to do, just add a note and say: Wish to learn in 2025, and revisit whenever you have time.

I guarantee you, that very important thing, the next time you see it, you might be happy you didn’t spend lots of time on it, as it wasn’t really something important.

Use something like the Eisenhower Matrix, where you identify the tasks into different buckets (Urgent + Important, Important but Not Urgent, Urgent but Not Important, Not Urgent + Not Important), where most stuff seems urgent in the moment, but irrelevant a couple of days or weeks later.

# Process

I just went through my NirvanaHQ todos, I cleaned up so many tasks I just added either for “sometime” or as next task, that I either did already without looking at the todos, or it became irrelevant, or it was just a note, something to keep in mind that I just copied into my notes and second brain, so I can continue to add related information and potentially use in the future.

But so many tasks, and task management should have more constraint, make it harder to add a task. So that we actually wish to open the Todo Apps, not to be afraid to be greeted with 1000 tasks, instead we know that we must open it to not forget something important. Or have the confidence not to open, and we know it will remind us.

So as always, it’s a balance. But that’s why I like NirvanaHQ, it’s a bit “old-fashioned”, you can’t add images or attachments, or you can’t make it fancy looking. But everytime I consider changing, I’m reminded that these constraints make it a strength, not missing features.

Also the workflow of GTD, is just baked in. I like that I don’t rely on reminder too much, but just on projects and status of next, later, schedule, waiting or someday. It makes it really easy to see most urgent (next) or just casual browsing through someday “todos”. or just seeing it of one project, or one area (business, personal) of life. Tags is another great filter option.

I think more is not needed anyway, otherwise we spend more time massaging the todos, than working on them.

Check out Getting Things Done (GTD) by David Allen, the book is great even though it’s an older system, and NirvanaHQ, which was built on that philosophy.

# Weekly Lists

See how I use Weekly Todo Lists.

# Work vs. Personal

I recommend having different tools for work and personal. Otherwise you will be stressed if you want to look at your personal tasks on your vacation, and then you see all your tasks that are waiting for you when you return to work.

Usually work gives you a tool anyway like JIRA, Azure DevOps or Project Management Tools, use them if you don’t have a favorite. I use Basecamp for my work, and NirvanaHQ for personal, especially long-lived todos. Not like buying this, or cleaning my room.

# Calendar

Involve a calendar. For scheduled tasks on a specific time or day, just put it in your calendar. This is what it is used for.

# Further Reads

Origin: Doing my todo’s today.
References: Personal Knowledge Management Workflow for a Deeper Life, PKM

Obsidian

Thu, 27 Aug 2026 13:45:38 +0000

Obsidian is a Second Brain and note-taking app. It is a powerful and extensible knowledge base that works on top of your local folder of plain text files.

Obsidian is a wrapper around your folders and Markdown files, including one single folder called .obsidian in your root. That’s it! In case you like to use another editor such as vim, VSCode or just any other Text Editor you might have, open all your files individually. Or, in case you want to open all notes in one go, use open Obsidian.

# The Power of Obsidian

I use Obsidian for all aspects of life, but not for everything. For example, I don’t it for todo’s or planning that have reminders/deadlines I must not forget. It’s not a todo app, and Todo’s need constratins. I have weekly todo lists, but it’s a different workflow.

But regarding notes, thinking, thoughts, ideas, doctor and health information, what I talked with friends, names of friends kids, they go all in one Vault in Obsidian. The power only comes, if you have a place where notes can be, and also be found years later. That’s where Obsidian shines (related Tweet).

# Second Brain

Not everyone needs a second brain is what Matt Jones is saying. To me I don’t mind how you call it.

In the old days, we called it notes app, or note-taking app. I call it second brain, as in Obsidian I can connect notes, which is after using it for a while, is quite the game changer, as I don’t relay on folders. And because I can find notes based on backlinks (e.g. a friend told me a tool or a quote, but I forgot the tool/quote, I just go on my note of my friend, and find the backlink).

Again, I don’t mind how you call it, but having a notes app is definitely something everyone should have. Start simple, add features only if you need them. KISS principle. I share more on My Obsidian Note-Taking Workflow.

# Why Obsidian

There’s so many reasons, using Vim Motions and using Markdown, locally, offline available and super fast might be the biggest.

# Markdown and Long-term

Whatever works for you, you should use. Don’t mind others. You should always try, IMO. If it doesn’t stick, don’t put in the high effort to move everything, only to move back in a year or so. The value from note taking is in the long term, so more important than your tool is your notes that stay with you.

That’s why Obsidian is great because it’s just Markdown files. Notion is proprietary, and as my 10 years of OneNote notes, which I could barely export to anything useful, besides the fact that my studies, my math, and all the valuable insights would have been there that I’d look up today, but can’t. As Kepano beautifully says, File Over App.

Some more videos and links in Why Obsidian.

# How I Create Links in Obsidian

I struggled with that as well. And I noticed over time, I automatically linked less. I like the example. Over time, you will change and update your notes and also feel when to use a connection and when it’s obvious.

Also, to relax your brain, e.g., I wanted to link everything to Obsidian as it was the root for many things I’m writing. But even though you do not link it, you will still see the connections in Unlinked mentions. This helps me not to think I will lose the note.

Usually, every note should at least be linked to one note. I also do that with my template that has an Origin and References. If I didn’t come up with a link through the text, I make sure that I at least link it to one reference that I think relates most, so it’s more likely to come up months or years down the line. Most of the time, the link also comes from Origin, as it’s usually a person or a blog that I already have in my Second Brain.

Here is the footer I add automatically to each note:

1
2
3

---
Origin:
Created <% tp.date.now(&

Sometimes, I even create an intermediate new note to the next, as it helps me to find and categorize my thoughts better.

But again, with time, you will figure that out; these come naturally. Especially when you check backlinks to a note, and you have 100s of them, they get a little useless. Therefore, you naturally add fewer in the future.

See also why quality software deserves your hard-earned cash and Local First approach. Find all my configs for Obsidian (hotkey, plugins, etc.) on my linked dotfiles.

# How to Link

From Obsidian: The Good Parts - YouTube

# Named Links

There is also Named Links (Obsidian), that you can give your link a name. Apparently full supported by the Obsidian:

I do similar things with Origin: LINK, that will tell that this link was the origin. But it does not show up on the graph:

Does it need to?

# How to Use Obsidian for Work

For work, keep in mind that meeting notes are the least valuable of them all, so don’t try to make them more than they are. Just keep one note with this structure like:

# Project / Person / Recurring meeting topic  
  
## Logs  
  
### Date 1  
Participants person 1, person 2, new also [[Person 5]]

discussed:  
- bla bla  
- Bla  
- ? why we need X  
  
Decisions made:  
- We use [[Project Z]] to optimize [[Problem YY]]  
  
### Date 2

And then make sure when you have some time, that you focus on the follow up, and potentially work on the note [[Problem YY]] by just clicking on it (will create a new note).

# Example

# Importig the Most Sold Book and Connected Book

A great example, wheter you read the bible or not, but as the bible is super connected all together, you can import it to Obsidian which then looks like this:

Nowadays, when I verse reminds me of something, I can now connect any of my notes with the actual bible verse with simple Obsidian wikilink and # for referencing a specific verse in a chapter.

This stars to be visible with the thin connections between mine and bible connection:

See the details in Import Bible to Obsidian (Markdown).

More Graphs

If you like the graphs, you can find more of mine in Obsidian Graphs.

# Price and Features

Let’s get this out of the way, Obsidian is entirely free! You can add two paid features, 1. if you want to use their integrated sync tool, or 2. if you want to share your notes, including the fancy graph and outline view.

But yeah, as it is plain text stored on your hard drive, you can use Dropbox, Google Drive, or any other sync tool! As I’m using Sync.com with end-to-end encryption for all my files, it does not work nicely on my mobile. That’s why I added Obsidian Sync. I could have added it to Dropbox, but I didn’t want to give them my most private notes.

If you want to check out all the other features, see the complete list at Obsidian.md or find more customization with 25 core and 329 community plugins, 60+ themes, plus custom styling, you can tweak Obsidian to work and look exactly how you want it.

Some community plugins do a single thing extremely well, like Calendar and Kanban. Others unlock endless possibilities: Obsidian Dataview and Templater are great examples.

See more on How to Take Notes in 2021?.

# Obsidian CLI

Obsidian CLI and Obsidian Headless with Obsidian Sync.

# Plugins

# Synchronisation

Use Obsidian Sync: Best and easiest way, but costs
or Obsidian Sync with Google Drive

# Four Modes of Writing

Four Modes of Writing

# Vim-mode

I use Vim Motions in Obsidian, they work very nicely.

Jump to next Markdown Heading with ]] and [[
toggle fold za ( .vimrc) and zR unfold all and zM to fold all
Vim for Obsidian
Vimrc for Obsidian
Vim Motions for Writers

# How I Search

I search by Quick Search / Omnisearch. I explain all of it in My Obsidian Note-Taking Workflow
- same does Kepano the CEO of Obsidian: Kepano search in Obsidian

# Zen-Mode

I wanted a «Zen Mode for Writing» to focus on Obsidian the same way I have in Neovim with <leader>z. So, I added a shortcut to my .vimrc in Obsidian to hide the right bar and tab bar. It’s simple but effective.

Video: Bluesky.
Configs in my dotfiles: dotfiles/obsidian/.vimrc at ed6239b202d703b98ab72235d6cfb784e363fe5e · sspaeti/dotfiles · GitHub
Neovim config, including Tmux bar hidding: dotfiles/nvim/lua/sspaeti/plugins/zen-mode.lua at ed6239b202d703b98ab72235d6cfb784e363fe5e · sspaeti/dotfiles · GitHub

# How to Get Started with Obsidian

It depends a little on what phase you are in (do you already have a note app, etc)? But generally, I’d start with plain Obsidian, don’t install any plugins. Just take notes and create a simple folder structure that makes sense to you (I’d add 4 folders to begin with -> Areas, Project, Resources, and Archive). And then you start using it. That’s it. You can always optimize later on.

The video I stole my initial folder structure from is from Zowie from Systematic Mastery; this video has great explanations if you want to know a little more: (old but still worthwhile).

If you want some more inspiration, I made a video about my Obsidian note-taking workflow: (or as a blog post). This is after 10 years of using One-Note and 2-4 years of Obsidian, so don’t try to replicate from the beginning if you just start out. My Obsidian Note-Taking Workflow

But the best thing about Obsidian is not Obsidian itself, but it’s open file format, Mardown. As the CEO likes to say, “File over app”.

# Explainer Videos

# Comparison to Apple Notes or Notion

Here’s my thought on someone wanting to go away from Apple Notes as it’s deleted half of her work.

If you want t e simplest solution for simple text, use Obsidian. Don’t start with plugins or customizing, just start using it and make an Map of Content (MOC) for your scripts, and link them back to it. So you don’t lose the overview. Obsidian can’t be simpler, because the files are local on your machine, so nobody is fiddling with or tampering with your data. You can work offline and use any sync tool (e.g., Dropbox, Drive, GitHub, etc) to back up.

It’s a million times faster than Notion, as it’s locally on your machine, no network required. The only thing it won’t work for is if you need to share or collaborate, that’s when you can use Notion, still, I’d say. But that’s why Obsidian makes the perfect “Second Brain” as it’s yours, and you don’t need to collaborate, just deep uninterrupted thinking.

Additionally, because it’s just local text files, you can even open it up with different editors, e.g., if you write a long article, or a book, or a deep note, you could open it with iA Writer or similar, to be even more focused. AND because it’s not proprietary files like Notion, Apple, and all cloud services, you can extend, upload to a website, and customize to all your needs. But that should be done later in the process.

# Why not Neovim?

I’m a heavy Neovim user, and I still use Obsidian for notes. I tried a couple of times using Neovim for note-taking, and sometimes, when writing my book, I use Neovim. But for the most part, Obsidian is optimized for notes specifically, and Neovim is optimized for programming.

With Obsidian based on Markdown, I have access to my Second Brain with forward/backlinks to any note or article. Better integration with images, text-based diagrams with Mermaid), and text-based images with Exaclidraw, and Canvas.

Integration with Plugins such as ReadWise syncing my highlights and notes I read, Obsidian Dataview to use Notes as databases, Admonition (Call-outs) to write lovely in-side comments without distracting the reading flow, Templates, Mobile support, and many more.

The key for me is to use Vim Motions within Obsidian but have all the optimized environment for note taking. I write more in My Vim-Verse: The Backbone of My Workflow which might be interesting to you, or My Obsidian Note-Taking Workflow (A Vim-Inspired Approach to Efficient Note Management with Obsidian and Markdown).

PS: Although I started using Neovim when I wanted to focus, get rid of all distractions, and strip my Neovim dotfiles to function as a word processor on my distraction-free typewriter. Originally posted on r/Obsidian

# Using Obsidian with AI

Keep AI Out of Your Obsidian Vault

# Misc

# Market

See StackOverflow Survey 2024, most desired besides Markdown. .

# Feature Wishes

It serves almost all my needs already; if I have to wish:

mobile opening up within 1-3 seconds, not 10.
smart connections with local-llm integration trained/aggregated with my vault notes
1. Second Brain Assistant with Obsidian (NoteGPT)
integrated OCR image search
respecting my existing Neovim configs for native vim navigation

# Optimize Obsidian Mobile

5 Killer Hacks to Boost Your Obsidian on Mobile. | by Lohith dhaksha | Obsidian Observer | Medium
Background battery optimization for obsidian and Brave/chrome:
- Initial Launch takes too long : r/ObsidianMD: I have about 2.5 GiB of data, in more than 6k notes and some hundreds or thousands of attachments. Obsidian takes less than 8s every time (8s is the slowest since 1.7.7), and usually between 5s and max to start.
- What I see that makes a difference is RAM memory and excluding the app from battery optimization so that it doesn’t get killed.
- Mine stays active for a day this way, so opening it – except for the first time – is instantaneous.
- Also, reindexing shouldn’t occur often. If it is, there’s something that’s changing your files metadata or creating too many of them while Obsidian isn’t running.

Reddit - Dive into anything:

If I am not wrong, if you set battery to unrestricted for both Chrome browser and Obsidian, it will stay in background. Read it somewhere.
I used to try this but then again, sometimes leaving it in background cause me more problems. Dataview not updated. Editor goes haywire. Eventually gave up on keeping app in background.
Then again, maybe it is just my phone problem.

More Obsidian Troubleshoot.

# On Linux

I use it heavily since 6 month or so in Arch Linux Omarchy. it works great, i think you need some parameters to start (I dont think they are a must, but there are some one Omarchy).

Sync, writing and all the plugins work as on macOS. i only have two issues:

I use vim mode, and 2 or 3 times a day, probably in combination with lists, I cant get back to normalmode and markuos are not rendering anymore. So I quickly restart Obisidian.
The second, less important, but still a pity, is that the graph view is seriously slow. it barely works with one neighbor level (I have 10k+ notes tough). But that worked much better on macOS. I thought is was the computer, but in the meantime, I have a ultra fast laptop, and its still the same.

# Technical

# Illegal Symbols

So there are basically two reasons why characters would be considered illegal by Obsidian:

links invalidation: []#^|
file systems limitation: \/: and ? for Android

Other than these most characters are legal, like those emojis being unicode characters, and most other unicode characters, like stuff with accents/diacritics: áāãőůçĉ, and national characters like æøåßþƴðđ.

So feel free to use characters rather freely. See also Internal links and special characters - #2 by holroy source

# Troubleshooting

Obsidian Troubleshoot

# Settings

All settings explained: The Obsidian Settings Guide I Wish I Had When I Started - YouTube

# Graph

See the history of my graphs at Todays Daily Graphs - Obsidian Graph.

# Further Reads

My Obsidian Note-Taking Workflow: Blog, YouTube
My 20+ years of note-taking summarized: Knowledge Management in the Digital Age: From Zettelkasten to Startup Owner | Obsidian Workflow (see also in My Gigs)
How I take notes: Structure with Now Next Notes | Sebastian De Deyne
Personal Knowledge Management Workflow for a Deeper Life — as a Computer Scientist
Different Types of Writers // Archetypes of Writers
Why I switched to using Obsidian (as a former Notion user)

Origin:
References: Second Brain, Todays Daily Graphs - Obsidian Graph
Created

How I write Daily Notes differently

Thu, 27 Aug 2026 13:45:38 +0000

Many are writing Daily Notes. I tried several times, but it never stuck. I think it’s because it’s too much work and too cluttered. For a look at my Daily Notes, there are some critical questions about gratitude. It lets me focus on yesterday’s todos and what I want to achieve tomorrow.

# Cluttering Effect and Todo’s on Paper

But I think it’s just too much work that I add on every day. I might add valuable things to my daily notes, but I feel like that information gets lost in many small daily notes instead of being used as a source for smaller, connected ideas. I could always create notes after or while typing and directly move the journaling notes there. But then, why am I doing it all together?

I noticed that daily Todo’s and planning for me worked best on paper, I use Weekly Todo Lists. When I see it on my desk and think about the task, I’m fully offline, which helps me already to think more clearly. If I am finished, I can check it off, and at the end of the day, I have feedback and see that I solved all tasks for today or directly what I need to do for tomorrow.

Also, I didn’t like the clutter effect. You flush your valuable notes with invaluable notes, creating a new one daily.

If Digital: Weekly Todo Lists

What I have discovered again lately: If I want to do it digitally, I do a weekly one, always at the same time. This way, the questions don’t get too repetitive, and I can use that weekly (a daily that I do once a week) for planning my whole week. Check more at Todos.

# Thoughts On Notes Instead

Instead, I use “Thoughts On Journal” notes. So, I might have Thoughts On-note on my dad, priorities, jobs, work-life balance, regrets, and many more. So whenever I want to journal, I put it directly in the most related.

A list of my thoughts on notes in my Vault.

In there, I have headers that relate to daily notes. So, in a way, I’m still doing daily notes, but I’m doing it the other way around. Instead of opening today’s notes, I open related Thoughts On-note that already existed. And link it back to the “date”, without creating the daily note or date file. But I still have the history or can track everything back to the day. If something happens on the same day, I have several notes pointing to that day. Also, I add the created day to each of my notes anyway; I also have that connection of when I made a new idea on which day, too. More on My Obsidian Note-Taking Workflow if that interests you.

Sometimes, I randomly want to ramble, and it does not fit any existing (though that is rare), then I just put it to the closest note, or if something is new, I create a new note with the title ... (Thoughts On) as seen above.

# Template for Thoughts On Note

I store the discussion under the person’s note and have a header for “Meetups” where I then have chronological the latest meetups, and if I like, I add some pointers and important things I want to remember. I might even connect something if it makes sense. The note would look like this:

# My Topic Thoughts on

## [[2024-08-03]]
Today I thought about my work-life balance, and that I work too much. .... 


## [[2024-02-01]]
I want to work only 80%, because I have time for doing more [[Creativity|creative]] things, leading to more happiness.

This way, I have only a few notes, and even better, when I read through them, I have the complete history on a particular topic, as with daily notes spread around, you’ll never read them anyway as they are so intertwined.

I got this approach from Derek Sivers; see Benefits of a daily diary and topic journals.

# Daily Notes on Persons/Friends/Family

I noticed that I do the same with meeting notes and various journals. I have one DevJournal for work, which includes what I do every day, but it is separated by headers (weeks and days) and simple bullet points.

But even more important to me, I use the same approach with friends and people I meet. I open their note (with the person’s name) and add a dedicated section, e.g., Meetups or discussions and protocol, which are the most important things I want to remember.

This looks like this:

# Persons name

who this person is, and how I met him. Maybe who he/she is married to, if they have kids, I put their names as I always forget them.


## Meetups

### [[2024-08-03]]
We ate at .. in Bern. He told me that he had a new company, but it didn't work out with his co-worker.

He bought a new keyboard.

### [[2024-07-03]]

Two years ago we met up again, we talked about ...

Instead of cranking everything into a daily note and pulling them together with Dataview, I write them directly to the person.

This has the same benefit as the Thoughts On Journal, which I have within one note—the complete overview of a person and the last conversations we had with topics that were important to me.

# Daily notes vs. Thoughts On Notes

I liked Andy’s reply where he likes Daily notes fertile for ideas and thoughts that span in a broad manner rather that in depth manner. Meaning in Daily notes he can be more horizontaly, thinking about anything that comes to mind as in Thoughts On notes you go vertically, deep into one topic while maybe blocking other ideas that come to mind.

Also similar to Work Log Definition and My Logs.

# For Gratitude and Reflections

What I learned too, is that for gratitude and reflections, it works best if Ic does that on a yearly cadence, but with a very long and excessive 🔃 Reflection Template. This way I can easily compare with last year and don’t tend to write the same over and over again, which happened when I asked for what I was grateful for yesterday as only so much can happen during a “normal day”.

# Conclusion

This was a short brain dump when browsing Reddit on Uses of the daily note, and it inspired me to write my methodic down. Also, this benefits me even more as my vault gets bigger.

Origin: Uses of the daily note : r/ObsidianMD
References: Obsidian, My Obsidian Note-Taking Workflow

The Vim Language (and Motions)

Thu, 27 Aug 2026 13:31:21 +0000

What are vim motions, and what is the vim language? In one sentence, it’s a grammar for editing text. The question is: why are so many developers and writers (including many new ones) still using this “old” technology called Vim or Neovim?

Besides the editor just mentioned, it’s mainly for its vim language/motions. Many believe, myself included, that you need to know 1000 shortcuts. But when I realized that Vim has a language that works like grammar, things changed. I noticed that I only have to remember a couple of modes (Copying (or yanking) in Vim with y, deleting with d, pasting with p, changing with c) and shortcuts, which I can then combine. This made me want to try it, and I didn’t regret it.

It surprises me that I use it even more for writing articles than for coding, which I think I will use it for. However, the efficiency of editing text with the precision of a surgeon is even more helpful when I edit and rearrange my blogs or books.

Vim motions, the vim language, its grammar: if you edit or write text (coder, writer, etc.) 8 hours a day, you probably need to learn it, as you did with touch typing.

# Vim Language

Vim has a terrific language or grammar behind its shortcuts. Instead, remember a thousand shortcuts; you can learn a couple and combine them. These are sometimes called the Vim language or Vim motions for moving around. This has nothing to do with the editor yet; they are universal and available in other editors as well.

For example, VSVim for VSCode, IdeaVim for the JetBrains products, Vintage Mode for Sublime, set -o vi in terminal, bindkey -v in tmux, and so on. But there are also Browser extensions like Vimium or Firenvim, and even Gmail adapted some of its shortcuts for navigation (j, k for moving, g for jumping).

Everyone who types on a computer eight hours a day should learn the Vim language. Yes, it’s hard in the beginning, but that’s everything new and different. getting better every day and having more fun coding or writing should motivate enough. The excuse of being too busy to learn while doing what you always did does not count.

Are you too busy to improve | Image from steenschledermann

Typical hotkey without vim-motions

Typical hotkey with none-vim setup. They are useful, but very limited as I explain in my Vim write-up, making the case why I have chosen vim years back.

Source Boilerplate see Writing is Thinking

# The Grammar

As grammar has verbs, subjects, and objects, so does the Vim language. The grammar has different verbs to begin with. Copying (or yanking) in Vim with y, deleting with d, pasting with p, changing with c.

For example, the easiest shortcut is copying a line with yy. In this case, yank is the verb and the second y is a synonym for y_. The y is doubled up to make it easier to type since a joint operation.

Next, we can add movements. Each verb takes a subject to their movements. There are lots of movements (more in the next chapter); the easiest is with numbers. E.g. to copy three lines, you add three in front, such as 3yy. You can do that with all verbs, e.g., deleting three lines 3dd. Another would be { and } to move to the beginning or end of the paragraph, respectively.

In addition to verbs and subjects, the Vim language also has objects. E.g., we can save text into different clipboards (called a register in Vim) with "ay. Here, we copy it into register a, which would be the object. We can paste it again by doing the same but using the verb paste instead of yank "ap.

If you will, there are even adjectives and adverbs with prefixes. Usually, you use a verb and an object. But instead of going down three lines with 3J, which joins the following three lines. You could add d5}, which means “delete from the current line through the end of the fifth paragraph down from here.”,

For me, the most considerable magic and myth about Vim. All of the vim language is how you navigate end edit text, and it still has nothing to do with the editor. Sure Vim was the one that introduced and perfectionized them, but as listed above, you can get them anywhere else. This goes deep into the Vim language, yet we still need to touch the editor. This is important to know.

I hope you start seeing the power of such a pattern. With a couple of verbs and objects, you can already know hundreds of combinations without memorizing them. You can see a video on Mastering the Vim Language on the Vim language or read a full plea for the Vim language on a terrific StackOverflow comment.

# Vim Motions

As touched in the Vim language. Vim motions are how you navigate, whether you navigate to the end of the word or back to the start of the document; these are all motions.

These are the first thing you start learning (and hating) when you start learning. It’s extra hard to know initially, but something you want everywhere when you get used to it. Instead of using arrow keys, Vim uses jk to move down and up and hl to move left and right. The main idea is to use the keys that your right hand naturally rests on. You do not need to move any of your hands or even fingers for navigation. Again, this seems like a small thing, but once learned, you know why everyone is telling you that.

Some common ones are:

h,j,k,l - left, down, up, right
w,W - to start of next word or WORD
b,B - to start of previous word or WORD
e,E - to end of word or WORD
$   - to end of line
^   - to start of line

You find the most important ones to start in this cheatsheet:

Vim Command Cheat Sheet from Cloud Guru

# Modes (normal, insert, visual, command)

Modes are what get you confused at the beginning. When you launch Vim, you are not typing what you click on your keyboard as you are not in the “insert” mode that you know. Instead, the normal mode you are in lets you do the commands explained in the above Vim language and motions.

Vim is the only editor that optimizes editing text instead of writing from a blank page.

Three modes illustrated (escape mode being the command mode) | Image from Geekforgeeks

That’s another reason Vim makes you so efficient: you have different modes for each phase of your task—normal mode for reading code and navigating quickly. Insert mode when you want to add some code or text. Visual mode is unique, the same as highlighting text with the mouse, but with the above Vim motions. The command mode is the powerhouse, where you can type Linux commands such as formatting a JSON file with :%!jq (whereas jq is a command line tool installed on your machine) and execute them within Vim. As well as Vim commands such as :sort for sorting your file.

I could go on here, but I want to dive into the editor itself now and explore why I learned it initially and how to get started if you’re going to. In case you want to read more, I wrote a full article on Why Vim Is More than Just an Editor and also about My vim-verse: The Backbone of My Workflow.

# Vim motions for none-technical people?

I got asked if vim motions worth for none-technical people, especially with a none-admin account at work? Below is my answer I gave.

First, I would focus on the “vim motions” (the shortcuts) rather than the editor itself. You might even be able to use these shortcuts at work, as there are now Vim extensions for almost everything. Unfortunately, not yet for MS Word or other similar programs.

What you can certainly do is write Markdown locally (which works on any operating system) and find an editor where you can install vim extensions (Obsidian, VS Code, Sublime, Notepad++, etc.). There are many tools that you can launch directly from a file (portable version).

Whether it’s worth it is difficult to say. If you write a lot as an author, and this is your profession, then definitely yes. I actually need it more as an author than for programming. It’s absolutely the best way to write or edit text. You could also use an online tool like HackMD, which also has vim motions that you can activate.

Switching between your personal computer and work computer shouldn’t be an issue, because you’ll never forget normal writing (without vim), and whenever vim motions are available, you’ll just be much faster. The only thing is that it will take you longer to learn the shortcuts and the vim language, and for these to become muscle memory.

Another important question is the keyboard layout. Some time ago, I switched to the US layout because vim motions are best designed for it. But if you don’t program, this doesn’t play such a big role, and you probably will have fewer problems if you stay with the Swiss layout.

I would just try it out. There are many cool exercises like Vim Golf or Vimtutor. It will be challenging at the beginning. And you’ll quickly notice whether it doesn’t suit you at all, or if it might be something for you. After 2-3 weeks, I started to enjoy it, and it became really “fun” to edit something with vim, even if the task itself would otherwise be very boring.

# Good Design Matters

As the vim motions has been here for many years, and integrated in any editor, any tool, it’s passed the Lindy Effect, and we can say, it has very ergonomic design.

Like professional StarCraft players don’t use the mouse for not its speed, but also they can just use their muscle memory, the same vim users use vim motions because of its speed, and Thinking in the thought of speed. Interesting how it connects to Thinking, Fast and Slow (Daniel Kahneman), where slow thinking is advocated.

Source by Boilerplate YT Video

More on this very interesting topic also by No Boilerplate YT on his video about Writing at the Speed of Thought.

# Further Reads

Article I wrote related to vim motions, and how I use them:

Origin: Why Vim Is More than Just an Editor
References: My Vim Cheatsheet, Vim Motions for Writers

My KBDFans DZ65RGB Built

Thu, 27 Aug 2026 08:20:29 +0000

Parts:

Kit: KBDFans DZ65RGB DIY Kit
Keycaps: Cheese DYE-SUB Cherry Profile
Switch:
- First: Glorious Panda
- Since 2025-04-11 upgraded to Drop Holy pandas, the original. And they are supriour in sound and feel ( little harder but so nice.) I lubed them immediately.
Lube: Krytox 205g0

# Built in 2021-02-06

In the process of building the individual parts together:

Comparison with Ducky One 2 SF:

# Upgrade

Putting in real holy pandas, not copies and relubing

# Sound Test

2026-08-27 more accurate - such a beautiful THOCK, no? 😉
How to update Second Brain with Obsidian—My Workflow with Quartz - YouTube

Older sound test:
Soundtes in 2025-04-21 after relubing and Holy Pandas (?) upgrade:
Your browser does not support the video tag.

Origin: My KBDFans DZ65RGB, original posted on Imugur, but got deleted

My Public Second Brain Workflow with Quartz

Thu, 27 Aug 2026 08:14:39 +0000

This is my documentation of how I publish my notes from a private Obsidian vault to my Public Second Brain (this very site) with open-source Quartz and GoHugo.

The source code, including all scripts, themes, and processes, is available at GitHub Second-Brain-Public.

Long-Form YT-Video

Part of how I publish is also on my YouTube video My Obsidian Note-Taking Workflow about my Note-Taking Workflow article.

# My Journey of Publishing Notes

I had several iterations. I used to have an HTML/CSS/PHP website, moved to a traditional WordPress blog, and now have a Static Site Generators (SSG) blog with GoHugo and a Second Brain (where you read this note) with GoHugo/Quartz.

On this page, I elaborate on how I can simply write in my Obsidian Vault, add #publish anywhere in my note, and run make deploy and it’s published on my public brain. No conversion, no extra formatting, no nothing needed, just running a make file.

I can write offline, on my mobile phone, or wherever I want; Obsidian will sync it across. Publishing will happen on my laptop (by running make deploy, which essentially runs an rsync to my web hosting).

# My Publishing Process

How do I publish? There are two steps.

I added a Python script find-publish-notes.py to the Makefile to copy all my SecondBrain notes from my Obsidian Vault with the hashtag #publish and copy it into the Quartz git-repo.
1. Since 2023-06-07: I switched to a Rust-script that does the same but adds #hashtags properly as Hugo requests, and it is 30x faster. To get the Rust executable, you need to run cargo build --release from within the utils/obsidian-quartz folder.
I ran the make deploy that syncs the changes to my website. This command copies notes with the hashtag “publish” and deploys to brain.ssp.sh. See also more in below .
1. Since I use GitHub, I could also create a GitHub Action to deploy on commit. But I actually prefer to run the make file.
2. This make deploy also included a go-program that collects all backlinks into two JSON files assets/indices/contentIndex.json and /assets/indices/linkIndex.json. This small util used is hugo-obsidian (see Makefile) written by Jacky Zhao. Here’s the source. But it is not maintained anymore (as there is now a v4 without it) and it had bugs and didn’t show all my backlinks. That’s why I forked it and fixed the backlinks. You can find it here: sspaeti/hugo-obsidian.

Here’s a short video of how the worflow looks like in action:

I really love this easy way to publish your thoughts, notes, and blogs. Here's my workflow, from an Obsidian Markdown file to a second brain note with one command.

No conversion or formatting, no copying from a note-taking tool to the email client or Substack. The original… https://t.co/wizLrhhQI9 pic.twitter.com/JbIrAj0axC
— Simon Späti 🏔️ (@sspaeti) August 27, 2026

Quartz v3 Workaround

I copy all my Zettels in the root folder instead of adding sub-folders in Quartz. This way I can use Wikilinks [[]] and do not need to change to absolute paths - which I do not want to activate in my Second Brain.

# Deployment

Here I describe how I deploy my second brain.

# Self Hosting

I self-host on my own server as I already have hosting for my website. The only thing I do I rsync the public-folder to my web hosting, that’s all. The script shows how I do it. The actual line is rsync -avz --delete public/ USER@DOMAIN.com:~/www/brain (just replace with your user and domain).

# GitHub

If you do want to use GitHub actions, you can check the Data Glossary (glossary.airbyte.com) which is the same setup as Quartz, but there we use GitHub. Checkout the required deploy.yaml that makes the magic work

Basically, when someone changes the branch hugo, it will deploy changes to master branch and publish that. I think I wrote some more details here.

# Published Words and Notes

Currently published notes on my second brain (public with #publish) per 2026-08-02 are 660 notes and 308k words.

In my private second brain, not published are roughtly 2 million more words and a total of 7,500 notes.

                    notes          words
All                 8,359      3,066,397
Published             660        308,300
Private             7,699      2,758,097
Archive               116         75,210
Active              8,243      2,991,187

────────────────────────

Words categorized
mine   2,284,229
quoted  191,255
clipped 372,787
skipped 205,540

────────────────────────

                  files         size
Notes             9,620        27.4M
PDFs                 50        58.5M
Images            7,680         3.0G
Other                14         5.8M
─────

Also Published

I also published 103 blog posts on ssp.sh/posts with 329'934 more words and a book on dedp.online with 157,089 words written and 1/15th or so, published.

# Added Features

# Adding an Image (Front Matter)

I added the option to overwrite the header image, and its width and height for each note, as well as the description. E.g. in the Frontmatter I can now overwrite the terms with:

---
ogimage: de-vault.jpg
ogwidth: 1024
ogheight: 761
ogdescription: this is my new description
---

# Mermaid as OG:image

Newly added:

`1`	`ogimage: mermaid`

And it will search for the mermaid in the note, takes a screenshot and makes this the OG image of this note.

Full features and change log at README.md.

# Obsidian Integration

This Hugo site includes custom processing for two key Obsidian markdown features, allowing seamless migration of content from Obsidian to the blog.

# 1. Wikilinks `[[]]`

Converts Obsidian-style wikilinks to external links pointing to your second brain.

Source: /layouts/partials/process-wikilinks.html

Example:

1
2

[[Default Path]]
[[My Note|Custom Display Text]]

Output:

1
2

<a href="https://ssp.sh/brain/default-path/">Default Path</a>
<a href="https://ssp.sh/brain/my-note/">Custom Display Text</a>

Configuration: Set brainBaseURL in your site params:

1
2

[languages.en.params]
brainBaseURL = "https://ssp.sh/brain/"

# 2. Images

Converts Obsidian-style image links to Hugo imgproc shortcodes for automatic image processing.

Source: /layouts/partials/process-images.html

Examples:

1
2

![[img_Why are we here on Earth_1745240448324.webp]]
![[my-screenshot.png|Alt text description]]

Output:

1
2

{ {< imgproc src="img_Why are we here on Earth_1745240448324.webp" caption="">}}
{ {< imgproc src="my-screenshot.png" caption="Alt text description">}}

Workflow: Simply copy your images from Obsidian’s attachment folder to your Hugo post’s bundle folder, and the image references will work automatically.

# 3. Admonitions

Converts Obsidian-style admonitions to styled warning/note boxes using the theme’s admonition system.

Source: /layouts/partials/process-admonitions.html

Supported Types:

- Note admonitions (blue styling)
- Abstract/summary admonitions
- Information admonitions
- Tip admonitions (green styling)
- Success admonitions (green styling)
- Question admonitions
- Warning admonitions (orange/red styling)
- Failure admonitions (red styling)
- Danger admonitions (red styling)
- Bug admonitions
- Example admonitions
- Quote admonitions

Examples:

Standard format (with empty line):

>  Important Notice
> 
> This is a warning message that will be displayed in a styled box.

>  Additional Information
> 
> This is a note with additional context.
> 
> Multiple paragraphs are supported.

Compact format (no empty line):

>  Quick tip
>
> This content appears directly after the title without an empty line.

>  Achievement unlocked
>
> This format also works for single-line content.

Output: Styled admonition boxes matching the theme’s design system.

# 4. Quotes

Normalizes blockquote formatting by removing unnecessary <p> wrappers for consistent styling.

Source: /layouts/partials/process-quotes.html

Example:

`1`	`> this is a quote that does not look nice rendered`

Without processing:

`1`	`<blockquote><p>this is a quote that does not look nice rendered</p></blockquote>`

After processing:

`1`	`<blockquote>this is a quote that does not look nice rendered</blockquote>`

# Processing Pipeline

All Obsidian features are processed in sequence in the layout templates:

Files: /layouts/posts/single.html and /layouts/preview/single.html

{{- $step1 := partial "process-images.html" (dict "Page" . "Site" .Site) -}}
{{- $step2 := partial "process-wikilinks.html" (dict "content" $step1 "Page" . "Site" .Site) -}}
{{- $step3 := partial "process-admonitions.html" (dict "content" $step2 "Page" . "Site" .Site) -}}
{{- $processedContent := partial "process-quotes.html" (dict "content" $step3 "Page" . "Site" .Site) -}}
{{- dict "Content" $processedContent "Ruby" $params.ruby "Fraction" $params.fraction | partial "function/content.html" | safeHTML -}}

Processing Order:

Images: Convert ![[image.ext]] to Hugo imgproc shortcodes first (before wikilinks to avoid conflicts)
Wikilinks: Convert [[Link]] to HTML links pointing to your second brain
Admonitions: Convert > title to styled admonition divs
Quotes: Normalize regular blockquote formatting

This processing order ensures that all features work together seamlessly. You can use wikilinks within admonitions, and images are processed before wikilinks to prevent conflicts.

# Uploading Example

# Long-form Example of how I Publish to Quartz

Link: My Obsidian Note-Taking Workflow - YouTube

Short form in this tweet.

# Short Asciinema Video

# See Other Public Brains

Other Public Second Brains

Origin:
References: GoHugo Static Site Generators (SSG) Shared on SH: Hackernews, My Tech Stack

My Terminal Workflow for Note-Taking, Data Engineering and Writing (Linux/MacOS)

Thu, 27 Aug 2026 07:41:57 +0000

I have a very sophisticated workflow setup that I have optimized over the years. My dotfiles have over 200+ stars on GitHub and grew from macOS to Linux with Omarchy, which I use today.

Below is my personal terminal workflow showcase from different work perspectives as a data engineer, developer, and writer/note-taker, spanning macOS and Linux (Arch via Omarchy). It covers tiling window management with Yabai+skhd and Hyprland, Neovim and Obsidian for writing and data engineering, Vim motions for writers, database querying with vim-dadbod, note-taking with Zettelkasten and PKM workflows, and links to dotfiles, YouTube demos, and Asciinema recordings.

There’s more around my second brain, but this is a short overview of some that come to mind right now. I might add more over time. Click the images below to get to the full YT video walkthroughs.

# My Different Workflow Showcases

# Omarchy (current Linux Setup)

Omarchy (using today) video about Arch Tiling Window Workflow (macOS comparison):
Omarchy Arch Tiling Window Workflow (macOS comparison) - YouTube

You also find it in writing at Arch Linux (Omarchy) — 8 Months Later: The Good, the Bad, and the Fixable (or why I switched in Part 1)

# MacOS

MacOS Tiling Windows Setup for Data Engineers & Writers with Yabai + skhd (used before Omarchy):

macOS Tiling Windows Setup for Data Engineers & Writers | yabai + skhd - YouTube

# Database SQL Query Workflow

Query databases in vim (vim-dadbod):

# Computer Desk Setup and Workflow with Monitors and Virtual Desks

Computer Desk Setup and Monitor Workflow with monitor alignment and virtual desktop setup:

Find some more workflow videos at my Asciinema page.

# Screen Alignment and Virtual Desktops

Computer Desk Setup and Monitor Workflow
Throughout this journey, I’ve tried various alignments and configurations, ultimately concluding that the best setup for me is a single screen with separate desktops. I switch between them using these handy shortcuts:

🖥️ Super+1: Terminal for coding
🖥️ Super+2: Browser for research and browsing needs
🖥️ Super+3: Obsidian for notes, thinking, writing, and my Second Brain
🖥️ Super+4: Music, featuring Spotify and Focus@Will for all things auditory
🖥️ Super+5: Collaboration tools like Slack, Asana, Nirvana, and my to-do lists
🖥️ Super+6: Extra (when laptop screen is open): Additional screens for meetings with Zoom, etc.
🖥️ Super+7: Extra (when laptop screen is open)

I’ve also experimented with various keyboards and mice. Currently, I’m using the Kinesis Advantage 2 and My KBDFans DZ65RGB as my daily drivers.

What is your go to setup after trying different setups?

Swapped Alt with Super

I swapped to Super instead of Alt since I’m on Linux (and using alt for tmux). Check out Omarchy as my Linux setup and My Journey from macOS to Arch Linux with Omarchy. And I use now alt for tmux direct shortcuts, e.g. alt+d to jump to my dotfiles, alt+w for writing my articles, alt+b for my second brain, and so on.

# Monitor Alignment

Image Source

Which one are you? 😉

# How I Landed on a Single Screen Setup?

I found that I can’t focus on both documentation and example at the same time anyway, so having them in the optimal position in the middle, and then switching between them ultra fast is key (single shortcut, hence above ). I feel like these speed-reading tools, which always put the words in the middle at the exact same position, make it easier to read vs. constantly jumping left to right with the eyes and searching for the last point.

Also, there are tasks that I move my browser next to my terminal, which is why a 27’’ works well for me, as you can still put them side by side when needed. Or what I do much more, pop up a Claude Web as an overlay window, hit it off, and then hide it again. Only for video calls, I prefer to keep my laptop as a second screen so I can do stuff while sharing the screen.

Original inspiration I also got from ThePrimeagen, which is a huge single-monitor proponent.

# Setup for X (Writer, Data Engineer, Note-Taking)

# My Workflow Setup for Writers: Vim Motions

I’m a full-time writer, and I use Obsidian and Neovim heavily as my writer tools.

I have written about Vim Motions for Writers, especially Vim Motions is key, if you work for a living on a computer, as it’s made specifically for text editing.

In Vim motions for writers, I write about:

The Flow of Writing an Essay
and my My Writing Process
and I have a video of how this looks in action.

See it here:

Vim Motion for Writers Showcase (Inside My Obsidian Vault) - YouTube

I have multiple articles about this topic too, see essays about Vim and writing.

# Why I Use Vim a Lot

In Why Vim Is More than Just an Editor – Vim Language, Motions, and Modes Explained | ssp.sh I write:

I’m still using Obsidian a lot more for writing (see more on my PKM workflow) due to its additional features of supported images, backlinks, graphs, and plugins specified for note-taking, such as ReadWise) (syncing my highlight from books, and tweets), Dataview (using notes as a database), Excalidraw (drawing with Markdown format), Templates, and so on.

Nevertheless, I write more and more in Neovim. For now, I use ZenMode (for centering the text), Grammarly (for linting grammar), write-good (linting grammar), and specifically Obsidian.nvim (follow backlinks, and so on.). You find all details in my dotfiles.

In Obsidian, I use the Vim mode Obsidian .vimrc and map most hotkeys to my Vim settings. At the same time, I’ve been writing more and more in Neovim and have been progressively moving over to full Neovim.

Plugins such as Telescope and simple grep features that I use for coding work very well with Markdown. Here are some clips showing what is possible in an excellent talk about Writing, Editing, and World-Building at the speed of thought with Vim.

# Getting into the Zone: Zen Mode

# Data Engineering

Why Vim Is More than Just an Editor – Vim Language, Motions, and Modes Explained at the chapter “Vim for Data Engineers”.

# Note-Taking Workflow

My Public Second Brain Workflow with Quartz showcased:

I really love this easy way to publish your thoughts, notes, and blogs. Here's my workflow, from an Obsidian Markdown file to a second brain note with one command.

No conversion or formatting, no copying from a note-taking tool to the email client or Substack. The original… https://t.co/wizLrhhQI9 pic.twitter.com/JbIrAj0axC
— Simon Späti 🏔️ (@sspaeti) August 27, 2026

I wrote at length at:

Or in long form video, if you have time:

Link: My Obsidian Note-Taking Workflow - YouTube

# My Web Clipping and Web-Highlighting Workflow

I used ReadWise for a while, but I completely changed my workflow to Obsidian Webclipper.

When I have lots of highlights and I’m on the computer, I create a new note with Webclipper and copy it directly as a quote, or add notes/comments inside the Obsidian note, instead of highlighting on a web page and then syncing that into Obsidian, which would also work with Webclipper.

So not only did the tool change, but the workflow did too. Even more, what I like is taking clips of books, and with Obsidian Webclipper, it allows me to add comments directly; no need to wait until it syncs. Meaning, if I clip a book that I got recommended, I can directly add the person, or a link to where I got the recommendation from, into the note at Origin:. So when I actually get to what I want to read next, or if I discover the book after a couple of years, I still know who suggested it to me, which is important information that belongs to each book if I haven’t found it myself.

If you use Obsidian Webclipping, see my templates at My Templates, so you can use it for YouTube, LinkedIn, GoodReads, Receips, etc.

# Terminal GitHub Workflow in Neovim

Working with GitHub PR within Neovim

# Further Reads

Origin: tmux
References: Essential Linux Commands

Semantic Layer

Wed, 26 Aug 2026 11:36:26 +0000

A Semantic Layer acts as an intermediary, translating complex data into understandable business concepts for users. It bridges the gap between raw data in databases (like sales data with various attributes) and actionable insights (like revenue per store or popular brands). This layer helps business users access and interpret data using familiar terms, without needing deep technical knowledge.

The Semantic Layer serves as a translator between various data presentation layers (Business Intelligence, Notebooks, data apps) and data sources. It integrates data sources, models metrics, and connects with data consumers, translating metrics into languages like SQL, REST, or GraphQL.

A Semantic Layer defines key business metrics (like “active” users or “paying” customers) once company-wide, eliminating inconsistencies across different tools. This centralization and standardization of definitions ensure uniform understanding and reporting across the organization.

The Metrics Layer, a subset of the Semantic Layer, was first introduced by SAP BO in 1991. The Rise of the Semantic Layer provides more on this. Kimball Group defined it in 2013.

Headless BI vs. Semantic Layer

Headless BI, often used interchangeably with a Semantic Layer, can be considered a practical implementation of the latter. The term’s origins are traced to a LinkedIn Comment.

Insight from Maxime Beauchemin ( Podcast)

The Semantic Layer is like a restaurant menu: you know what you’re ordering, but not how it’s made. This layer maps metrics to physical tables and can range from minimal modifications (thin layer) to encompassing transformation logic atop physical tables (thick layer).

# Semantic Layer Definition

My definition goes something along the lines of:

A semantic layer acts as an intermediary, translating complex data into understandable user business concepts. It bridges the gap between raw data in databases (such as sales data with various attributes) and actionable insights (such as revenue per store or popular brands). This layer helps business users access and interpret data using familiar terms without needing deep technical knowledge. Universal Semantic Layer: Capabilities, Integrations, and Enterprise Benefits

Or by Julian Hyde and this is how SQL could express ‘measures’ to bridge the gap:

A semantic layer, also known as a metrics layer, lies between business users and the database, and lets those users compose queries in the concepts that they understand. It also governs access to the data, manages data transformations, and can tune the database by defining materializations.
Like many new ideas, the semantic layer is a distillation and evolution of many old ideas, such as query languages, multidimensional OLAP, and query federation. Building a semantic/metrics layer using Apache Calcite

See also Data Engineering Whitepapers, “Measures in SQL”.

# Why a Semantic Layer

In my previous company, we developed an Analytics API, similar to what Cube does, but with orchestration as a key component. We switched from SSAS to Druid, a Modern OLAP solution, to handle diverse business metrics queries from Tableau, Notebooks, and our Web App. The inability of Druid to store queries as SSAS led us to create this Semantic Layer or Analytics API. The Semantic Layer’s real advantage lies in its ability to define and automate metrics as code, offering a universal, open-source, or open-standard approach to handling business metrics.

More of my thoughts are elaborated in Building an Analytics API with GraphQL and in The Rise of the Semantic Layer: Metrics On-The-Fly. More Slack Convo.

# Example of How Semantics Look Are Defined

See Semantic Layer Measure Defininition Examples.

# Different Types of Semantic Layers

From What Is a Semantic Layer? | GoodData:

Semantic layer in a data warehouse: The main purpose of a data warehouse is to provide a centralized data source for the whole organization. It is designed to be a single source of truth for different departments, user groups, and use cases. The structure of data in the warehouse can be complex and technical, which makes it difficult for users to access the information they need. As a result, business users often extract portions of this data into BI tools, creating localized semantic layers that can contribute to semantic layer spread.

Semantic layer within data pipelines: When constructing data pipelines (the process of adding data from various sources to a data warehouse), data engineers input a semantic layer in the code. This layer helps to name and organize the different parts of the data models, such as tables and attributes.

Semantic layer in Business Intelligence (BI) and data analytics: This type of semantic layer defines business concepts and the relationships between them. It also defines metrics and calculations that can be used for analysis and reporting through different users and user groups for specific business use cases.

Universal semantic layer: There is a connection between raw data and the different tools for users to analyze their data (such as BI and AI/ML tools, management tools, and business applications). A universal semantic layer doesn’t focus on a specific business use case and needs to cover company-wide requirements.

# History

The Evolution of the Semantic Layer (and related for context MDM, dbt/Jinja):

1990: (SAP) BusinessObjects (France) launches Skipper SQL, introducing the BI semantic layer concept (later called the “Universe”). The Universe Designer tool became a named product in the v3/v4 era (1994–1996).
1997: SSAS and MDX with their logical modeling layer with MDX, define business metrics and dimensions in a structured way (1997)
2008: Master Data Management (MDM) (with MDS from Microsoft in 2008)
- Business entities (customers, products, locations): MDM focuses on core business entities (customers, products, locations) while semantic layers typically focus on metrics and dimensions.
- Single Source of Truth: MDM with master data records and SL for Metrics
- Data Governance: Both approaches involve managing and governing data definitions
2013: Kimball discussed the concept of a semantic layer in #158 Making Sense of the Semantic Layer
2016: Maturing BI tools with an integrated semantic layer such as Tableau, TARGIT, PowerBI, Apache Superset, etc. have their own metrics layer definition
2018: Jinja templates and dbt eroding the transformation layer into a semantic layer
- Not by definition, but the dbt declarative SQL definitions, defining all your DWH, in a way is an early semantic layer. In a way creating single source of truth (although potentially many different 😅).
- Like if you think what the old SAP BO Universe was, it was a logical model of SQL definitions. In a way, dbt definitions are the same. You do not have the visual designer, except you run dbt docs. That’s at least my thought and how it relates to the history overall.
- I think the line is fine. You can define measures and dimensions in dbt as SQL and add stuff with Jinja, but maybe too far stretched to call it semantics. BUT, it is declarative with SQL :) Full Discussion
2019: Looker and LookML popularized as the first real semantic layer
2022: Modern Semantic Layer, Metric Layer or Headless BI tools such as MetriQL, MetricFlow, Minerva, dbt arose with the explosion of data tools (BI tools, notebooks, spreadsheets, machine learning models, data apps, reverse ETL, …)

more on The Rise of the Semantic Layer | ssp.sh

# Semantic Layer Tools

Cube
dbt Semantic Layer (MetricFlow: now part of dbt)
MetriQL
GoodData
AtScale with their Universal Semantic Layer
Snowflake Semantic View
Databricks Unity Catalog Metrics
Malloy
Looker with LookML
Honeydew: Semantic Layer specific for Snowflake
Lightdash’s integration: Lightdash semantic layer
Rill has a Metrics Layer built in
Small and light once:
- Boring Semantic Layer by Julien Hurault
- sidemantic by Nico Ritschel: A universal metrics layer. Compatible with definitions in LookML, MetricFlow, Cube with DuckDB, Snowflake, Clickhouse, Bigquery & more.

# Integrations

Open Semantic Interchange (OSI) (new name Apache Ossie): A attemp of large vendors to unify semantic models as open-source, vendor-agnostic specification. The specification uses MetricFlow as reference implementation.
Sidemantic: The universal metrics layer. Compatible with 15+ formats: Cube, MetricFlow, LookML, Omni, BSL, LDM, Cortex, Malloy, OSI, SML, TML, Hex, Rill, Superset by Nico Ritschel
- A semantic layer: define metrics once, query them consistently across tools. Sidemantic is a semantic layer: a system for defining models, dimensions, metrics, and relationships once, then querying them consistently.
- e.g. run it with will with git clone https://github.com/sidequery/sidemantic.git && cd sidemantic and uv run examples/rill_demo/run_demo.py
  `

# Choosing the Right Tool

Burak Karakan said on LinkedIn:

If you have nested data, Malloy

If you have tabular data and advanced relations/joins: Cube

If you need something easier and only DuckDB-based, I would just stick to DuckDB Macros: CREATE MACRO Statement – DuckDB

I have done this, and it was just fine for simple use cases.

I keep the macros in .sql files as the source of truth and push them to GitHub. Then, I use Lorenzo Mangani’s webmacro extension to load them into DuckDB memory at runtime.

The benefit of Semantic Models being just SQL files is that you can have static data and can implement your JOIN logic or measures in a native SQL way. In my opinion, this is superior to any YAML/Python approach since it removes the middleware, which makes things less transparent and customizable.

Then we have the duckdb-fastapi adapter to expose metrics in a REST API for applications. For integration with BI tools and SQL clients, we expose DuckDB via the Snowflake wire protocol, which enables any tool to access metrics via SQL.

# When not to Use a Semantic Layer

Don’t use a semantic layer when you start out. A user said it will slow down innovation.

True, a (modern) semantic layer is usually not something you start with, except you use it as federated, universal API for your distributed backends (multiple DBs, source systems etc), but the more common one is once you have a larger organization where KPIs are not defined across, or people don’t know where to get the “correct” definition.

But when you start out, use SQL, MVs, dbt, and persist all the tables. Or build your semantics in an OLAP cube or BI tool; if you only have one, no need to add an extra tool.

In Why Semantic Layers Matter — and How to Build One with DuckDB, I explain that the simplest and most straightforward reasons are:

You’re just getting started with analytics and only have one consumer, meaning you only have one way of showcasing analytics data, for example, a BI tool, notebooks, or a web app, but not multiple ways of presenting data. This means you don’t apply calculated logic in different places.

You don’t have extensive business logic that you query ad hoc; you have simple counts, SUMs, or averages.

You preprocess all your metrics as SQL transformations into physical tables, meaning your downstream analytics tools get all metrics preprocessed and aggregated, and filtering is fast enough.

# Semantic Layer Defintions

How do you define metrics and dimensions in a semantics layer, you might ask.

See Metrics and Semantic Layer Definition for some examples with Cube, Boring Semantic Layer and others mentioned above.

# Use Cases of Semantic Layer

From the great talk by Brian Bickell from Cube: Semantic Layer Deep Dive w/ Brian Bickell (Cube)

# Similar to `MVC`?

Idea

Could the Semantic Layer be likened to the MVC (Model View Controller) model? In Cube (OLAP), views are created similar to DB views, with the model and component handling the rest. This is akin to Convergent Evolution between Semantic Layer and MVC.

Wrote more about this on Exploring the Semantic Layer Through the Lens of MVC - Cube Blog.

# Knowledge Graphs, LLMs with Semantic Layer

Semantic Layer as the Data Interface for LLMs
Natural Language for SL

# Why not Define Measures within SQL?

This is what Julian Hyde brought up in his talk Extending SQL for analytics, similar to what MDX Studio did to SSAS.

# Other Resources

# Pedram Navid

Post LinkedIn / Pedram Navid on LinkedIn: #dbt #metrics | 38 comments: While dbt is building a metrics layer, the question still remains whether a metrics layer outside of BI will ever gain wide enough adoption. Jacob Matson rightly points out that Looker, Thoughtspot, Power BI, and Transform all have a metrics layer tightly integrated within BI, and they are good enough.

The challenges dbt has are that its implementation is pretty bad (no one wants to write Jinja Template in yaml), it lacks features critical for it to be useful (like joins), and it’s not clear that we need widespread access to metrics across tools outside of BI.

# Michael Driscoll

replies on The metrics layer may not actually need to be a layer, it could get baked into the SQL standard. (Extending SQL for analytics)

Databases could implement it. And every BI tool could query metrics directly.

Metrics layers are just aggregate expressions with some metadata.

Let them live in SQL.

If DuckDB Labs can introduce ‘GROUP BY ALL’ and proclaim that ‘FROM foo;’ is valid SQL, surely they could bring us aggregate awareness too.

# Artyom Keydunov & Pavel Tiunov - Cube

Semantic Layer and its relation to MVC (Model View Controller) pattern, popularized in Ruby On Rails.

The concept of a Semantic Layer shares similarities with the MVC (Model-View-Controller) model, particularly as popularized by Ruby on Rails through its Active Record pattern. In a conversation with Artyom Keydunov & Pavel Tiunov from Cube.dev, Artyom drew parallels between the two:

Just as the MVC model focuses on decoupling data, the Cube Semantic Layer serves a similar purpose. In Cube, this decoupling is evident in how they create views, akin to database views, while the model and component handle the rest.
The Semantic Layer can be likened to the Active Record in Ruby on Rails. Active Record is an implementation of the ORM (Object-Relational Mapping) pattern, which abstracts and simplifies database interactions. Similarly, the Semantic Layer abstracts complex data structures, making them more understandable and accessible to business users.
At its core, the Semantic Layer represents the Logical Data Model in Data Modeling, serving as an intermediary between raw data and its representation to end-users.

This comparison underscores the universality of certain design patterns across different domains and technologies. Whether it’s web development with Ruby on Rails or data engineering with tools like Cube, the principles of abstraction, simplification, and decoupling remain consistent.

# Abhi Sivasailam: LookML is Still the Best SL?

LookML was the GOAT and every other “modern” semantic layer spec is a step backwards. How is this possible? Abhi Sivasailam

I believe if you only need to integrate into one BI tool, which LookML did, it’s much easier compared to open source. They must integrate not only in multiple BI tools, but also with notebooks, web apps, and even Excel. But in terms of syntax and implementation, they made many good decisions.

To me, the perfect “not so modern” semantic layer was the SAP BO universe. Still one of the best implementations to this day, but again, it still only had to support one BI tool.

# More Perspectives

A semantic layer represents business data in a way that end-users can access using common business terms. demistify metric
The Metrics Layer, synonymous with a Semantic Layer, was previously just random queries in BI tools. RW Deep Dive What the Heck Is the Metrics Layer

Explore more about the Semantic Layer:

Down the Semantic Rabbit Hole
The Missing Piece of the Modern Data Stack
Deep Dive: What The Heck Is the Metrics Layer
The Great Data Debate by Atlan
The Metrics Layer has Growing up to do
The Universal Semantic Layer, More Important Than Ever
Demystifying the Metrics Store and Semantic Layer
What Syntax for the Semantic Layer? - by Benoit Pimpaud
The Semantic Layer Is Back. Here’s What We’re Doing About It. | Preset
Lyft with a combination of Metrics Layer and semantic with YAML: Metric Semantic Layer: How Lyft Governs and Scales Key Data Definitions. Similar to what we’ve built with ValueStore with Metadata Store with SQLite.
An Agent’s Field Guide to Semantic Layers | Motley by Egor Kraev

I wrote a deep dive into The Rise of the Semantic Layer | ssp.sh, and Why Semantic Layers Matter—and How to Build One with DuckDB | ssp.sh, as well as a three part series:

Origin: Metrics Layer
References: The Rise of the Semantic Layer, Modern Semantic Layer, Ontology

Graph Databases

Wed, 26 Aug 2026 09:03:12 +0000

A graph database is a database that uses graph structures for semantic queries with nodes, edges, and properties to represent and store data. A key concept of the system is the graph.

# Databases

Neo4j
ArangoDB
OrientDB
Dgraph
AllegroGraph
GraphDB
Amazon Neptune
Stardog
TigerGraph
CosmosDB
JanusGraph
DataStax
GraphBase
Apache Giraph
Apache Jena
Nebula graph
Titan
Virtuoso
KuzuDB

# Based on Object Storage

OmniGraph
HelixDB

# Query Layers

Google Spanner Graph

# State of Graph DBs

Source by Andrew Altshuler in 2026-08-25

Origin:

Laptop/Notebook & Desktops for Linux (Hardware)

Wed, 26 Aug 2026 07:45:29 +0000

Hardware, desktops or notebooks optimized for Linux.

# Manufacurer

Framework Laptops
TUXEDO Computers - (The Tongfang is the same as tuxedo’s, but cheaper) / Tuxedo Laptops - I have the My Tuxedo InfinityBook 14 AMD
System76

# Laptops:

Good Linux Hardware · Omacom · DHH suggestions by DHH - updated 2026-08-13:

- Dell XPS 14/16: This is my favorite laptop at the moment. It’s sporting Intel’s superb Panther Lake CPU, so its got Apple M-chip levels of performance and efficiency, combined with a stunning tandem OLED screen that pops like nothing else. The build quality is also extremely good, and it’s very light weight. Battery life in excess of 12 hours is possible. Get the 358H w/ OLED screen.
- Framework 13 Pro: Wonderful keyboard, excellent 4:3 screen aspect ratio, and a battery that goes forever. This too features the Intel Panther Lake chip that’s changed the game for modern Linux laptops. It’s also entirely repairable. The nerd’s nerdy computer. Love it.
- Even Marques Brownlee reviewd it, and had only good things to say: Framework 13 Pro: The Modular Laptop is Real! - YouTube
- And look how it looks:
- Dell XPS 13: Dell’s answer to the MacBook Neo betters it in almost every way. 120hz matte screen, check. Backlit keyboard, check. Great single-core performance from Intel Wildcat CPU, check. Its US launch price of $699 makes it the best deal on the x86 side of computing. Really good keyboard too! But you’ll want to have a more powerful machine to SSH into if you’re a developer working on heavy multi-core builds.
- Tuxedo InfiniteBook Pro 14: Great matte 3K screen, very large 80wh battery (a third larger than the Framework 13!), dual NVMe drives, built-in ethernet + HDMI, 5xUSB ports, and all in a package weighing just 1.45kg. Only nit is the awkwardly small right SHIFT key on the US ANSI keyboard option.
- Asus Zephyrous G14: Amazing 2.8K OLED display that pops in a way no regular IPS or micro-LED screen can match. Comes with a variety of NVIDIA GPU options, which gives it serious performance for both gaming and smaller local LLM models. Build quality is a notch above Framework and Tuxedo, and on par with a MacBook, while still being user-serviceable for changing SSD, Wi-Fi card, and battery .

Lenovo Pro equivalent, **Lenovo ThinkPad X1 Carbon Gen 14 (Aura Edition) ** is the direct match — same Intel Panther Lake Core Ultra chip generation as the Framework 13 Pro and Dell XPS 13, in the 14" 16:10 form factor

Other Findings:

ROG Zephyrus G14 - David Gasquez
Lenovo with A Developer’s Review of a Snapdragon X Laptop (Lenovo Yoga Slim 7x) - Wesley Moore
Framework | Framework Laptop 13 Pro: Intel Core Ultra 3 & LPCAMM2 with Intel Core Ultra (x86)
Dell XPS vs. Framework Pro: Its Finally Here: The Framework Pro 13 - YouTube
ThinkPad Core Ultra 7 268V with 32GB of LPDDR5 RAM?

Panther Lake == Intel Core Ultra Series 3

Panther Lake" was Intel’s internal codename during development, and the official retail name is Intel Core Ultra Series 3.

# Suggestions

Buy a Linux laptop? Laptops with Ubuntu or Linux Mint | Laptopwithlinux
Reddit user choosing ThinkPad
Check Best Laptop Deals for recommendations.

# Mini Desktops

From DHH at Good Linux Hardware:

Beelink EQ mini Wildcat: Very respectable option for less than $500, comes with 12GB RAM + 512GB NVMe.

Beelink SER10: Great performance from the HX470. More than enough for almost anyone!

Framework Desktop: This tiny little beast packs the insane AMD 395+ MAX chip, which with up to 192GB of shared memory, 16 CPU cores, and 40 GPU cores, is just a mad power house of a computer. And yet it’s only 4.5L and completely silent. You’ll pay for the privilege at current RAM prices, but if you just want the best, this is it!

One more:

Minisforum

Origin: @davidgasquez.com on Bluesky
References: My Laptops or Notebooks, My Favorite Linux Laptops so far

Zoxide

Wed, 26 Aug 2026 07:42:32 +0000

A smarter cd command. Supports all major shells.

It remembers which directories you use most frequently, so you can “jump” to them in just a few keystrokes. zoxide works on all major shells.

GitHub - ajeetdsouza/zoxide: A smarter cd command. Supports all major shells.

# Add a Folder and All Subfolders Recursively

1
2

fd -t d -0 . ~/Simon/Sync | xargs -0 -I{} zoxide add {}
fd -t d -0 . ~/Pictures | xargs -0 -I{} zoxide add {}

Now I can Fuzzy find in yazi and have all folders available, even if I did not cd into them.

# Add with Higher Prio

Or, to make it higher prio, add multiple times - Reload shell, then re-seed Sync with higher weight (loop add → higher rank, survive aging):

for i in 1 2 3 4 5; do
	fd -t d -0 . ~/Simon/Sync | xargs -0 -I{} zoxide add {}
done
fd -t d -0 . ~/Pictures | xargs -0 -I{} zoxide add {}

# Configs

# SQLite DB

Stored at: /home/sspaeti/.local/share/zoxide/db.zo

# Paths

1
2

export _ZO_MAXAGE=1000000  # 100x default, room for many dirs
export _ZO_EXCLUDE_DIRS="$HOME/.cache/*:*/.git/*:*/node_modules/*:*/.venv/*:*/__pycache__/*:*/target/*:*/dist/*:*/build/*"

# Help


A smarter cd command for your terminal

Usage:
  zoxide <COMMAND>

Commands:
  add     Add a new directory or increment its rank
  edit    Edit the database
  import  Import entries from another application
  init    Generate shell configuration
  query   Search for a directory in the database
  remove  Remove a directory from the database

Options:
  -h, --help     Print help
  -V, --version  Print version

Environment variables:
  _ZO_DATA_DIR          Path for zoxide data files
  _ZO_ECHO              Print the matched directory before navigating to it when set to 1
  _ZO_EXCLUDE_DIRS      List of directory globs to be excluded
  _ZO_FZF_OPTS          Custom flags to pass to fzf
  _ZO_MAXAGE            Maximum total age after which entries start getting deleted
  _ZO_RESOLVE_SYMLINKS  Resolve symlinks when storing paths

Origin: zsh shell

Neomd

Wed, 26 Aug 2026 06:18:26 +0000

Neomd this is my way of implementing a email TUI based on my experience on Neomutt focus on Neovim (input) and reading/writing (Markdwon) and navigating, Vim Motions.

Keyboard-first TUI email: write in Neovim, render as Markdown, screen senders first, organize emails once.

eomd is a terminal email client that lets you compose and read emails in Markdown using your $EDITOR (neovim), rendering them as minimal but beautiful HTML on send with possibility of all Markdown features and Callouts — all from a single bubbletea TUI. It features a HEY-inspired Screener workflow where new senders are triaged into accept/reject lists before hitting your inbox, turning email into a GTD-friendly, and triage emails only once workflow.

see almost all feature: reading newsletter directly in your email client (feed) - see spy pixel, if you have replied (dot) and thread mode if replied)

Reading view on the right, different screener list below:

Find a short rundown on YouTube:

# What is Unique about Neomd?

The key here is speed in which you can navigate, read, and process your email. Everything is just a shortcut away, and instantly (ms not seconds). It’s similar to the foundations that Superhuman was built on: it runs on Gmail and makes it fast with vim commands.

With the HEY-Screener, you get only emails in your inbox that you screened in, no spam or sales pitch before you added them. Or don’t like them, just screen them out, and they get automatically moved to the “ScreenedOut” folder.

With the GTD approach, using folders such as next (inbox), waiting, someday, scheduled, or archive, you can move them with one shortcut. This allows you quickly to move emails you need to wait for, or deal with later, in the right category. Processing your email only once.

With the additional Feed and Papertrail, two additional features from HEY, you can read newsletters (just hit F) on them automatically in their separate tab, or move all your receipts into the Papertrail. Once you mark them as feed or papertrail, they will moved there automatically going forward. So you decide whether to read emails or news by jumping to different tabs.

# Core Features

Only a few of the features:

Markdown compose — write in $EDITOR (nvim default), send as multipart HTML + plain text
Attachments — via yazi or <leader>a in nvim; images inline, 1–9 to download
Preview emails including images.
Open HTML emails such as Substack or others with w3m
Multiple from addresses — SMTP-only aliases, cycle with ctrl+f
Glamour reading + Kanagawa theme — emails as styled Markdown, nvim palette
Markdown formatting with bold, links, and italic. Formatted in-line and code blocks, header h2, h3
Clever and sophisticated shortcuts to move or screen emails.

Find all at GitHub - ssp-data/neomd: A minimal email TUI where you read with Markdown and write in Neovim. · GitHub.

# How to Use

Read the details with all its shortcuts at GitHub.

flowchart TD
    Start([New Email Arrives]) --> AutoScreen{Auto-Screener
Known Sender?}

    AutoScreen -->|screened_in.txt| Inbox["📥 Inbox (Next)"]
    AutoScreen -->|screened_out.txt| ScreenedOut[🚫 ScreenedOut]
    AutoScreen -->|feed.txt| Feed[📰 Feed]
    AutoScreen -->|papertrail.txt| PaperTrail[🧾 PaperTrail]
    AutoScreen -->|Unknown| ToScreen[❓ ToScreen]

    ToScreen -->|Press I| ClassifyIn[Add to screened_in.txt]
    ToScreen -->|Press O| ClassifyOut[Add to screened_out.txt]
    ToScreen -->|Press F| ClassifyFeed[Add to feed.txt]
    ToScreen -->|Press P| ClassifyPaper[Add to papertrail.txt]

    ClassifyIn --> Inbox
    ClassifyOut --> ScreenedOut
    ClassifyFeed --> Feed
    ClassifyPaper --> PaperTrail

    Inbox --> Process{Process Email
GTD Decision}

    Process -->|< 2 min?
Do it now| Action[Reply/Handle
Immediately]
    Process -->|Waiting for others
Press Mw| Waiting[⏳ Waiting]
    Process -->|Not now, later
Press Mm| Someday[📅 Someday]
    Process -->|Time-specific
Press Mc| Scheduled[🗓️ Scheduled]
    Process -->|Delete
Press x| Trash[🗑️ Trash]
    Process -->|Reference only
Press Mp or P| PaperTrail
    Process -->|Newsletter
Press F or Mf| Feed

    Action --> Done{Done?}
    Waiting --> Review[Review Later]
    Someday --> Review
    Scheduled --> Review
    Feed --> ReadLater[Read in
Feed Tab]
    PaperTrail --> SearchLater[Search when
needed]

    Done -->|Yes| Archive[📦 Archive]
    Done -->|Not actionable| Archive
    Review --> Archive
    ReadLater --> Archive

    classDef folderStyle fill:#54546d,stroke:#7fb4ca,stroke-width:2px,color:#dcd7ba
    class ToScreen,Inbox,ScreenedOut,Feed,PaperTrail,Archive,Waiting,Someday,Scheduled,Trash folderStyle

all colored boxes represent neomd folders

Key principles:

Screener first: Unknown senders never clutter your Inbox — they wait in ToScreen for classification
One-time decision: Once you classify a sender (I/O/F/P), all future emails from them are automatically routed
GTD processing: Emails in Inbox are processed once — if < 2 min, do it or keep it in inbox as doing Next otherwise move to Waiting, Someday, or Scheduled
Minimal filing: Only Archive when done; no complex folder hierarchies — use search to find old emails
Separate contexts: Feed for newsletters (read when you want), PaperTrail for receipts (search when needed)

# History

Find the full history at Twitter - inspired by seeing Email.md on HackerNews.

It’s fully Vibe Coded, I only omitted the taste with pointing it to all my tools and astetics, also showing it the code from open source tools.

# Inspiration

See inspirations for this project such as:

Neomutt — the gold standard terminal email client; neomd reuses its screener list format and borrows many keybindings
HEY — the Screener concept: unknown senders wait for a decision before reaching your inbox
hey-cli — a Go CLI for HEY; provided the bubbletea patterns used here
HEY-Screener in (Neo)Mutt: My way of using HEY screener in neomutt.
Newsboat — RSS reader whose O open-in-browser binding and vim navigation feel inspired neomd’s reader view
emailmd.dev — the idea that email should be written in Markdown
charmbracelet/pop — minimal Go email sender from Charm
charmbracelet/glamour — Markdown rendering in the terminal
Kanagawa — the color palette used for the inbox
msgvault — Go IMAP archiver; the IMAP client code in neomd is adapted from it

If you rather see the prompt, check out initial prompt and its plan.

# Integrations

# It Works Suprisingly Well on a Mobile Phone

Not that I would use it all day long, but to browse my newsletter emails, or checking the emails, it’s really nice on a phone with Terminus app:

The Screener is running on my home-server

So that emails get automatically screened out and moved to ToScreen folder, I run Neomd as a Headless Daemon Mode on my homeserver - syncing emails every 2 minutes.

# Omarchy Plugin

Now also as Omarchy plugin womarchy-neomd-plugin to browse the emails right there from the top bar.

Full video at This Tweet.

# Roadmap

Some ideas in random order:

reminders for email if not replied yet only.
takes a while or hangs sometimes on Wifi or internet connection change
light mode shoudl have different text color (optimize for light mode or toggle automatically when light background)
When I reading an email: shortcut for listing all emails from that domain (and previous emails - similar to thread view, but based on domain, email instead of subject name)
Sent later feature: https://neomd.ssp.sh/docs/sending/#send-later
contacts.csv from google read only integration https://neomd.ssp.sh/docs/configuration/#contacts
store lists on IMAP?
- https://www.manpagez.com/man/1/alpine/ - similar to what Alpine/Neomutt is doing? (see Issue on github)
- optional - local first and offline should still work - sync between them?
emoji reaction - should we add a json attachment, gmail does that like this: { "emoji": "👍", "version": 1 } with the name attachment.vnd.google.email-reaction+json
extend serach of emails with DuckDB - see Full-Text Search with DuckDB - peterdohertys.website
- also check msgvault
people can add their own folder: make it dynamically? (currently hard coded) - see FAQ – neomd Documentation
offiline mode (?): this is a workflow change and atm not priority, but downloading offline to batch reply on an airplane would be great. Would need encryption too, that we download all emails encrypted. Potentially encrypting our lists too? download with POP3
- T4T5 on X: “@sspaeti the thing i really want is scriptability i’m thinking that if that if my data is stored as simple plaintext files, i can easily ask agents to write useful workflow scripts i’d like to scan my emails for calendar events to put in caldir, receipts to store in beancount, etc” / X

# Images in Neomd

image rendering with neovim Image API: https://github.com/neovim/neovim/issues/30889 (would be only during composing, not reading)
- not inside goldmark, as this does not work. Matcha uses sixel + kitty + iTerm2 + ghostty via terminal env-var detection, renders out-of-band by writing to stdout with cursor positioning (bypasses bubbletea). Uses mattn/go-sixel + stdlib image/png. and does not use glamour: has its own HTML→element pipeline (view/html.go, 1087 LOC).
Inline image rendering · Issue #20 · ssp-data/neomd
- 👀 goldmark-images: Goldmark image replacer extension · GitHub
! quickshell (e.g. https://github.com/sspaeti/omarchy-neomd-plugin) CAN SHOW images! maybe we could use that? to be checked out

# DONE

signature per-account
- (see Issue on github)
check similarity with matcha TUIs
Get notifications for certain domains/senders on Hyperland (integrated in notifications)
approving full domains, not individual senders only
Emoji reaction should be sent from the account the email was sent too
I use Listmonk for my email, it’s open source too, so I’m thinking of integrating an email or an integration that let’s me send directly from neomd to all my subscribers.. 🤔
! deamon to run on NAS or server, to automatically run screener to have screened emails also when opening on mobile
- Background sync daemon — separate neomd-sync binary (or cron job) that runs IMAP fetch + screener headlessly, so the inbox stays clean even when neomd is closed; complement to the in-process 5-min tea.Tick sync already running while neomd is open
- ? Should we use syncthing to sync text-files between my desktop, server and android phone?
(I just use normal spellchecker, works well)
- integrate Aspell (or any other spellchecker) into neomd - here someone integrated it into neomutt: Still loving the Micro Journal Rev.2 with Neovim—Chill writing session. : r/writerDeck
- has normal spell checker from neovim. not working super nicely yet, but it works: dotfiles/nvim/.config/nvim/lua/sspaeti/set.lua at a66451e7f29afce97f44d0965d31c0fa39d29c61 · sspaeti/dotfiles · GitHub
set up IMAP with proton.me and mail bridge https://proton.me/mail/bridge
Add callout support
- https://github.com/VojtaStruhar/goldmark-obsidian-callout
- https://gitlab.com/staticnoise/goldmark-callout
💡 Feature: Emoji reaction reply · Issue #9 · ssp-data/neomd
Showing and being able to attach files
Change A for switching account not a
cmd mode like vim wtih :
fix search? (e.g. find all keys that are shortcuts like above a, or disable shortcuts when searching?)
integrate Newsboat into Feed of my email? -> not great idea
Unread counts in folder tabs — e.g. Inbox (3)
having a signature when composing an email
d / u half-page scroll in reader (vim-style)
Reply-all — group reply to all original recipients
CC field in compose
Forward email — f key, pre-fills “Fwd: " subject with quoted body
Add an Attachment, quite important: the flow should be easy from within Neovim as this is the composer
choose sender in composer: setup gmail, hey, and others in one account to send - without need to switch account
highlight threads or conversations that are forth and back in a thread
Docs instad of readme.md?
better search: overall and keeping search when to apply multi command on search result
integration with contact or email - reading from screened_in list?
create a demo account for showcasing the features (e.g. neomd.demo@gmail.com)j
- create a hostpoint demo account, as gmail is way slower… defeates neomd
- when finished or restart
  - ./scripts/reset-demo.sh welcome screen
  - add or create truncates the screener lists (doesn’t delete them) and removes the welcome marker.
always on bcc in config.toml (if each email should be sent to your other email for example)
### Not priority right now
Background sync daemon — separate neomd-sync binary (or cron job) that runs IMAP fetch + screener headlessly, so the inbox stays clean even when neomd is closed; complement to the in-process 5-min tea.Tick sync already running while neomd is open
looks pretty nice, looking forward to that keychain support merge, or if you supported command line argument to fetch password like neomutt that would also be great PASS=pass xyz Reddit suggestions

Full Change log at neomd/CHANGELOG.md at main · ssp-data/neomd · GitHub.

# Presenting and Key Features

Points to showcase, see the video here - Neomd: Vim-native Email TUI with Markdown Compose & HEY Screener | Take 2 - YouTube:

opening new account
- welcome screener
- using neovim, yazi
- kanagawa theme
email start in inbox
- amount of loaded emails
screener
- and screen-all if you want
- GTD
- tabs
move around, move emails
- speed
- lists
- sorting
! search - search overall
reading emails: markdown
- opening w3m, browser AND nvim
- images
composing
- markdown native
- images attach
- send forth and back in neomd
- signature
- Preview
- drafts
- different [senders] and [accounts]
- autocomplete emails
saving drafts - UNDO
threaded view and reply DOT
- threaded view with T
comparision neomd with IMAP web view -> Always in sync if accessed from mobile
! everything folder
zoom out and jump to next read email
keyring
theming
ai integration, as it’s just a local markdown file
(Calendar integration)

# Further Reads

Email and Email TUIs
Email, and the way we (should) communicate at work | ssp.sh
Learning from someone else’s vibe coded project: Im going back to writing code by hand | k10s devlog
Gmail Thinks I’m Stupid, So I Left | moddedbear.com

# Reading RSS Feed vs. Reading Newsletters in Email

Reading in Neomd (Email) vs. Newsboat (RSS):

Simon Späti 🏔️ on X: “Reading in Neomd (Email) vs. Newsboat (RSS). https://t.co/ViIEfEFhIF” / X

# A Happy Neomd User

Just for fun, a happy neomd user 😉.

Generated with AI Meme Generator - Create Viral Memes in Seconds | Glif

Origin: Vibe coded after seeing Show HN: Email.md – Markdown to responsive, email-safe HTML | Hacker News

Kinesis Advantages 2: Lubing and Dampening

Tue, 25 Aug 2026 21:34:00 +0000

After a Ducky One 2 SF and then my first build, a KBDFans DZ65RGB, I have to say my first impression wasn’t good. The keys are wabbling, the switches are scratchy, and the sound is very hollow. And of top, the software to program any keys is crap. You can’t program a combination of two keys mapped to one, and there is essentially no second layer that you can shift with a key while holding, only toggle until you toggle it back.

Luckily after lubing the Cherry MX brown switches and filling the keyboard with kitchen towels (no joke), it’s getting better and better every day and especially getting used to the ergonomic keyboard, which was the whole reason for buying it. I’m still down at a couple of 40 words per minute, but accuracy and comfort are very high. I’m looking forward to using it for a while now.

I guess the next layout must be the same as, most probably after getting used to it more, there is no way of going back to a standard layout. At least, that is what I read here as well.

This is My Kinesis Advantages 2:

# Lubing

# Dampening with Kitchen Towels

Improved dumpening the sound

Inspiration for Kitchen Towels

For dampening, I used this: Kinesis Advantage2 LF Keyboard: My Custom QWERTY Layout, Touch-Typing, Dampening, Switches

Posted my first day review on Reddit

My post on Reddit: new keyboard: kinesis advantage 2 : MechanicalKeyboards

# More Images

# Update 2024-03-30

# Soundtest Afterwards

Your browser does not support the video tag.

Origin: Kinesis Advantage 2
References: My KBDFans DZ65RGB

My Mechanical Keyboards

Tue, 25 Aug 2026 21:33:30 +0000

My KBDFans DZ65RGB: Custom build, see full built
My Kinesis Advantages 2: Ergonomic (Find How I Lubed and Dampered the keyboard)
Ducky One 2 SF: First buy
My Distraction-Free Typewriter (Micro Journal) as of 2025-02-27
My Typewriter (Hermes 2000)

# Images

Hermes 2000 on the left, Kinesis Advantages 2 on the bottom right, and Micro Journal top right:

KBDFans:

Ducky:

Find them in various places in my Computer Desk Setup and Monitor Workflow images.

# Further Reads

My Mice-Mouse

Origin:
References: Dotfiles, Computer Desk Setup and Monitor Workflow

My Distraction-Free Typewriter (Micro Journal)

Tue, 25 Aug 2026 21:21:05 +0000

I finally got my baby, a Distract-Free Typewriter. I chose the Micro Journal Rev.2. Thank you, Un Kyu Lee. As a writer and author, this thing is a beauty and just what I didn’t know I wanted.

# Images

Split Screen: Writing my next article while looking up:

Micro Journal Rev.2 is out in the open:

Excursion to the local coffee house:

The spell checker on and off.

Chill session writing about my favorite topics, Business Intelligence:

Or in combination with an actual typewriter, My Typewriter (Hermes 2000), and the Kinesis Advantage 2 (find my Kinesis Advantages 2 Lubing and Dampering mods too 😀):

Which one is your favorite? 😉

View from the back, setting up and plugging in the power bank:

# Gallery

Find more at Todays Office images.

More Links to images and posts

My Bsky Thread

My Reddit Post with Neovim: Micro Journal Rev. 2 with Neovim as Word Processor : r/writerDeck, and How I’m stil loving it

Still loving the Micro Journal Rev.2 with Neovim—Chill writing session. : r/writerDeck

# Soundtest

Your browser does not support the video tag.

# Modifications and Configurations

I spent all day yesterday modding and updating the SW to make it my perfect writer tool. As a data engineer and author, I use Vim Motions all day long. On my laptop, I use Obsidian for writing and Neovim for programming. As the Micro Journal has limited memory and CPU, I decided to set up Neovim as a word processor.

# Features

Vim Motions as navigation and writing!
extensibility with multiple plugins.
- E.g., I can switch to different themes (light if I’m outside).
- I can split screens and open multiple files, jump back and forth, fuzzy find other files, link to my second brain with [[]], Markdown rendering (see headers ## are highlighted with colors, bold and links, lists, code blocks (!!), callouts are highlighted, and many more)

# What I Have Updated so far

yazi as a file navigator (a faster, Rust-based file navigator, the same as Ranger). It is opening much faster than Ranger.
Installed Neovim
I installed git to sync my writing, e.g., my book is written in Markdown. This way, I can simply continue on my Laptop when I finish with the Micro Journal.

I will definitely continue to tinker with it. Some of the Neovim configs are on wp.ssp.sh - it’s not the latest, I will update soon. But be aware that installing the latest version of Neovim was not straightforward. If you are interested, I can write more about it.

P.S.: I bought the Akko’s clicky Creamy switches and added KAM Playground keycaps. Next up, lubing the switches so that they are not that scratchy :)) Reddit, Bsky

# Why Not a Phone-Like Device

Initially, I wanted an e-ink device that could have apps. I wanted one that could run Obsidian, and there are devices that do all of it. But then I thought that it was just another “phone” like device, with all the distractions. So, with the limitations that come with a terminal-only device, I thought that’s just right.

Open writing on tty1 and with alt + F2 open another terminal and have Second Brain open there for research. Switch between them with alt F1 and F2.

# Modification Choices

I went for Micro Journal Rev2, see Micro Journal.

I went with

KAM Playground – Oblotzky Industries ( ~~Rome keycaps: nice but is cherry, not XDA!~~)
- The search of Keycaps and colors. Moved to Keycaps research for Micro Journal.
Akko V3 Pro Creme Blaue
Anker Power Bank, Powerbank 20000mAh 30W for charging

Convert neovim into a type writer or word processor: NeoVim as a WordProcessor. Example with emacs: My Microjournal Rev.2.ReVamp : r/writerDeck.

# Full Spec

Device: Micro Journal - Above version is a Rev.2 by Un Kyu Lee
Hardware: Raspberry Pi Zero 2W
Keyboard: 48-key ortholinear mini keyboard with side knobs for scrolling
Switches: Akko V3 Pro Creamy Blue (clicky)
Keycaps: KAM Playground by Oblotzky Industries (KAM uniform spherical profile)
Keymap Firmware: QMK / Vial
Battery: Molicel INR 18650 Li-ion
OS: Debian/Raspberry Pi OS based
Word Processor (Editor): Neovim configured as a word processor (Vim Motions, themes, split screen, fuzzy find, Markdown rendering, [[wikilink]] support)
File Manager: Yazi (Rust-based, replaces Ranger)
Version Control: Git — syncs Markdown writing between device and laptop

# Scripts

I also created a bash script for local file sharing via WiFi AP. No external WiFi required. I can share files between my phone and microjournal anywhere without needing to log into work, coffee shop, or home WiFi. My Microjournal Rev.2.ReVamp : r/writerDeck

#!/bin/bash

# Function to run when Ctrl + C (SIGINT) is detected
cleanup() {
    echo "Disabling NetworkManager.service..."
    sudo systemctl stop NetworkManager.service
    echo "NetworkManager.service disabled."
    exit 0
}

# Trap Ctrl + C (SIGINT) and call the cleanup function
trap cleanup SIGINT

sudo systemctl start NetworkManager.service
sleep 10
sudo nmcli device wifi hotspot ssid microjournal password microjournal ifname wlan0
echo "########################################"
echo "wifi:microjournal pwd:microjournal"
echo "Open http://$(hostname -I | awk '{print $1}'):8080 in your web browser"
echo "Ctrl + C to exit"
echo "########################################"

# Start the file browser, suppressing output
filebrowser -r ~/microjournal/documents -a 0.0.0.0 --noauth -d ~/file.db > /dev/null 2>&1

# Wait indefinitely until Ctrl + C is pressed
while true; do
    sleep 1
done

# Turn on/off Wifi

./startnetwork.sh starts wifi
./stopnetwork stops it

These scripts are under the microjournal (HOME) folder.

# Keymaps Keyboard Remap

The custom keymaps I made for my Micro Journal.

# Layer 0 - Base

# Layer 2 - DOWN (Fn2)

# Layer 1 - UP (Fn1)

# Layer 3 - CAPS

Inspired by the layout of My Kinesis Advantages 2 (as also ortholinear and limited space side wards)

Z and Y is Swapped to US Layout

I grew up with QWERTZ (mainly german speaking countries use), but switched to US layout, but kept the Z/Y out of habit :)

# Change Key Maps with Vial

Back up any writing on your Rev 2. This is just a precaution. Very little can go wrong here.
Shutdown the Rev 2 and click the power switch to off. Close the lid to ensure you don’t mistakenly use keys on the Rev 2 during this.
Connect the USB port on the side of the Rev 2 keyboard to your computer.
1. You should not disconnect this cord for any reason before two minutes has passed (just to be safe).
2. ~ Connecting the cable (use a USB-A to USB-C (USB-C to C does not boot!)) will cause the Rev 2 to boot.
3. ! If the cable is disconnected during the boot cycle it could corrupt the SD card.
4. Two minutes is probably more than you need to wait, but I’m being super-safe here.
Make your changes to either replace or move functions as detailed above. Click on the key you want to replace on your Key Map in the top half of the screen and then click on the function you want to replace it with in the bottom half of the screen. Repeat this for as many changes as you wish.
When you have completed all your changes:
1. simply exit Vial on your computer and
2. disconnect the USB cable that connects your Rev 2 to the computer.
3. The Rev 2 will turn off, but you won’t see that because the lid should be closed.
That’s it. The next time you boot up, you should have the changes on the Rev 2’s keyboard.

# Default Mapping

Default mapping of Micro Journal

Hooks-Keymap-Guide.pdf

# Install Neovim

If you want to install Neovim - neovim-latest.sh:

# Make sure snapd is installed and up-to-date
sudo apt update
sudo apt install snapd
sudo reboot  # You'll need to reboot after installing snapd

# After reboot, install the core snap
sudo snap install core

# Now install Neovim
sudo snap install nvim --classic

There’s also scripts for installing other libraries and tools I use such as ripgrep, zoxide and other tools:

install-wp.sh
or yazi: yazi.sh

# Troubleshoot

# Battery Completely Drain

I forgot to turn off the screen over night, now the battery is totally empty. I have charged the last 30 to 60 minutes now, but it is still not turning on. Should I be worried, or does that just take time?

When a lithium-ion battery like the Molicel INR 18650 is completely drained, it may require some time to reach a minimum charge threshold before the device will power on. This is normal behavior and a safety feature with many devices that use lithium-ion batteries.

Answer

In my case, it took around 65 minutes or more until two green lights were on, and then it was working again. Before it was only a red light on. Also, check if it’s charging well (there should be a red light on the MJ that indicates charging. Also, the charging only works from a USB-A to a USB-C (low power))

# Key Switch `|` Didn’t Work Anymore

I had to replace it with another, and then it worked again.

Cassidy Williams had similar issues described on Debugging my Micro Journal:
She also had the issue that the switches that weren’t working were just the furthest right two columns of keys! Googling this proved to be nearly impossible, until I found a Reddit comment buried in an old post:

There’s sometimes a glitch when powering the device with the left-side USB-C that’s farther from the keyboard, where some keys don’t work. The left-side USB-C that’s closer to the keyboard doesn’t have this issue.

# Using Obsidian on Low Battery Device?

I asked the Micro Journal creator Un Kyu Lee whether Obsidian would work initially:

I also use obsidian. But I don’t see a use case in the micro journal that directly invovles obsidian. Unfortunately, micro journal revisions are all low power computers and they don’t have enough computing power to run obsidian.

I would use micro journal to write a text and then do the obsidian management in the PC. But I understand that the power of obsidian is in the tagging and that would be missing in this type of workflow.

Probably, best is to use with a phone. I use obsidian on the phone and sync well with the PC using my NAS in the middle.

I might be releasing something that connects with the mini PC with a ortholinear keyboard. Maybe that revision may fit your needs.

I tried, and even Neovim takes a while to open, so Obsidian is definitely not possible with Electron background.

# Size: Expand to full 32 GB SD card

Only 3.2 GB was used, instead of full 32GB.

`1`	`sudo raspi-config`

Then select Advanced Options -> Expand Filesystem. That extended the partition to 32 GB.

# Potential Hardware

# Requirements of Obsidian

1 GB RAM minimum

# Other Distraction-Free Typewriters

Distraction-Free Typewriter

# Troubleshoot

# Neovim Stuck

# Solution

Go in and open another virtual terminal tty2 with alt + F2.
Run ps aux and kill neovim and yazi (I had to kill the parent process of yazi ps -ef | grep yazi) with kill -9 PID. Only then it was released.

# TTY

try Fbterm: for symbols and more colors compared to tty terminal: GitHub - sfzhi/fbterm: Framebuffer terminal emulator

# Further Reads

Uni-taskers
More about Linux and Omarchy where setup similar tools: My Journey from macOS to Arch Linux with Omarchy (Also check out Part 2)
My Mechanical Keyboards

Origin: Distract-Free Typewriter
References: My Typewriter (Hermes 2000)

Newsletters vs. Blogs: And Why I Still Blog

Tue, 25 Aug 2026 13:32:40 +0000

What I don’t like about a real newsletter is that you almost need to announce the new issue the day before so people can subscribe; otherwise, they miss the email.

In a blog, you can focus more on the writing, and you just publish. And Google and RSS make sure people read it. But on the other hand, in a newsletter, the already subscribed people get it for sure, so there’s no Death of the Follower happening as you’d share in the social or even on Google Search as they show up AI summaries before your blog post.

Just a thought that came up today while seeing a newsletter announcement (Substack note) with no value other than, tomorrow is a new send out, which, to be honest, I also sometimes do, but I really don’t like. I’d rather just share the link. That’s maybe Why I Still Blog, and I use my second brain for notes.

# Further Reads

Sharing as Second Brain Note vs a Blog Post

Origin:

Apache Parquet: Free Open-source Column-oriented Data Lake File Format

Tue, 25 Aug 2026 11:42:42 +0000

Apache Parquet is a free and open-source column-oriented Data Lake File Format within the Apache Hadoop ecosystem. It operates similarly to RCFile and ORC, other columnar storage file formats in Hadoop, ensuring compatibility with most data processing frameworks associated with Hadoop.

# History

Apache Parquet was officially released on 13 March 2013, marking a significant advancement in the efficiency of data storage. The format was designed to offer an optimized and flexible solution for handling large data volumes typical in big data scenarios.

Initial release

Chapter I: The Birth of Parquet | The Sympathetic Ink Blog

A noteworthy article by DuckDB Labs details Parquet’s significant role in modern data management: 42.parquet – A Zip Bomb for the Big Data Age - DuckDB

# Technical Benefits

Parquet is distinguished by its efficient use of Columnar storage, which minimizes I/O operations and enables better data compression ratios and encoding schemes. This format is particularly beneficial for analytical queries that process a substantial number of rows yet access only a subset of columns.

# Applications and Ecosystem

Parquet is widely adopted in various big data tools and frameworks, enhancing data interoperability and performance across diverse ecosystems. Its integration into platforms like Apache Spark and Hadoop and tools such as Pandas and Apache Arrow exemplify its versatility and robustness in handling complex data operations.

This entry ensures a comprehensive understanding of Apache Parquet, emphasizing its historical context, technical advantages, and broad applications in data engineering.

# Tools

Hardwood: A parser for the Apache Parquet file format, optimized for minimal dependencies and great performance.
Petquar: Visualize Parquet by Vu Trinh

# Newer Alternatives

Nimble and Lance from Nimble and Lance: The Parquet Killers - by Chris Riccomini.

Origin:

Second Brain: Crafted, Curated, Connected, Compounded on

(HEY) Email Screener

# What’s an email screener?

# Alternative Implementations

# Superhuman

# Further Reads

HEY-Screener in (Neo)Mutt

# So what is a Screener?

# How is this different from a spam filter?

# Use-Cases

# How I use it

# Screener Setup

# Setting up Neomutt Offline

# OfflineIMAP

# Config

# msmtp

# Neomutt

# Updated with notmuch

# Resources

Everything I've Published Came out of One Obsidian Vault

# What I Have Published and Written so far in Numbers

# From Note to Published; the Process

# Compounding Effect on Note-Taking

# How I Deploy and Share My Notes and Insights

# Shared Words Online and Private

# Full Breakdown of Numbers

# My PoDE Book Progress

# Data Views

# Over time with git (DEDP book)

# Further Reads

Compounding Note-Taking

# Further Reads

Notes Compound, Blogs Compile: When I Update a Note vs. Publish a Post

# Notes Compound

# Notes and Essays: Compiler

# Shake it until New Patterns Emerge

# Paper vs. Digital

# Further Reads

Will AI Replace Human Thinking? The Case for Writing and Coding Manually

# Guidance on When to Use It

# Soulless

# Distraction

# Don’t Get Me Wrong

# AI’s Are not Intelligent

# Exercising a Skill

# Attention that You Don’t Lose the Muscle of Writing

# Other Opinions

# Paul Graham on Writing

# Nathan Baugh

# Ted Gioia

# Mitchell Hashimoto

# Andrew Ng

# Harry Dry

# Jason Fried

# David Perell

# Ezra Klein

# Kepano

# Cal Newport

# Packy McCormick

# Dan Abramov

# Tim Ferriss

# David Heinemeier Hansson

# Wes McKinney

# Yann Martel, “Should you use AI to write?”

# Uncle Bob Martin

# Strauss Zelnick

# Naval Ravikant

# Will It Replace X

# Writers

# Writing Will Always Be an Asset

# Context Windows

# Writing More like Chess

# Everything is Based on Language (AI)

# Take Smart Notes

# Writers in David Perell Podcast

# Data Engineers?

# Programmers / Code?

# SWEs?

# Image Generation

# How to Detect AI Writing