Second Brain: Crafted, Curated, Connected, Compounded on

Common Table Expression (CTE)

Wed, 08 Jul 2026 14:54:59 +0000

A Common Table Expression (CTE) is a temporary, named result set, which can be referenced within SELECT, INSERT, UPDATE, or DELETE statements. CTEs are also usable in Views.

Will AI Replace Human Thinking? The Case for Writing and Coding Manually

Wed, 08 Jul 2026 13:34:41 +0000

Learning to Think Again, and the Cost of AI Dependency.

There are so many (hype/boring) posts about AI coming out every day. It’s OK to use it, and everyone does it, but still learn your craft, and try to think.

Find Good Data Sets or Sources

Wed, 08 Jul 2026 09:18:25 +0000

Data sets are always useful for data engineering projects. Here are some listed that can help find one.

Data Sets

Some that offer great datasets:

Neomd

Wed, 08 Jul 2026 08:41:13 +0000

Neomd this is my way of implementing a email [[TUIs|TUI]] based on my experience on [[Neomutt]] focus on [[Neovim]] (input) and reading/writing (Markdwon) and navigating, [[Vim Language (and Motions)|Vim Motions]].

Semantic Layer

Tue, 07 Jul 2026 22:44:40 +0000

A Semantic Layer acts as an intermediary, translating complex data into understandable business concepts for users. It bridges the gap between raw data in databases (like sales data with various attributes) and actionable insights (like revenue per store or popular brands). This layer helps business users access and interpret data using familiar terms, without needing deep technical knowledge.

DuckDB

Tue, 07 Jul 2026 22:05:36 +0000

DuckDB is an in-process SQL [[OLAP]] database management system. It has strong support for SQL. DuckDB is borrowing the SQLite shell implementation. Each database is a single file on disk. It’s [[SQLite DB]] for analytical (OLAP) workloads, whereas SQLite is for a relational database. But it can handle vast amounts of data locally. It’s the smaller, lighter version of Apache Druid and other OLAP technologies.

ssp.sh Newsletter

Tue, 07 Jul 2026 14:02:07 +0000

Welcome to my newsletter. I’m Simon, a data engineer, author, and founder & freelance technical writer. This newsletter is my way of sharing my ongoing journey in data engineering, writing, and [[Learn in Public|Learning in Public]].

Promoting Your Work

Tue, 07 Jul 2026 07:34:02 +0000

When you have a book or something you work on, you always need to promote it. Usually, no matter how good it is, it needs promotions. If you are going with a [[Book Publisher]], e.g., for a book, they might do some, but the bulk will end up on you.

Listmonk

Tue, 07 Jul 2026 06:37:38 +0000

Listmonk is an open-source and self-hosted newsletter and mailing list manager. Performance and features packed into a single binary. Free and open source.

CV

Mon, 06 Jul 2026 20:20:47 +0000

CV: Simon Späti

Data Engineer, Technical Author, and Educator

Location: Biel, Switzerland

Languages: German (Native), English (Fluent), French (Basic)

Introduction

I am an accomplished data engineer and technical educator with over 20 years of experience contributing to the success of diverse organizations across aerospace, banking/finance, computer science, energy, insurance, internet/web, military/defense, and public administration sectors. As a hands-on developer of innovative solutions, plans, and strategies, I focus on growing businesses, optimizing products and services, and boosting productivity organization-wide. I am a dedicated, empathetic, and entrepreneurial data engineer recognized for my superior performance and contributions to corporate missions.

Data Engineering Acquisitions (2022-2026)

Mon, 06 Jul 2026 20:14:09 +0000

Consolidation in the [[Data Engineering]] market is happening quickly. Tools from the [[Modern Data Stack]] get unified into bigger [[Data Platform|Data Platforms]]. This note highlights the latest acquisitions across data engineering. It serves as an overview of the latest consolidations.

Entity-Centric Data Modeling (ECM)

Mon, 06 Jul 2026 13:51:36 +0000

A new term was introduced by [[Maxime Beauchemin]] . Entity-centric data modeling (ECM), is a new approach to data modeling for analytics that elevates the core idea of an “entity” (i.e.: user, customer, product, business unit, ad campaign, etc.) at the very top of things.

LTAP (Lake Transactional/Analytical Processing)

Mon, 06 Jul 2026 12:38:52 +0000

[[OLTP]], [[OLAP]], [[HTAP - Hybrid Transaction or Analytics Processing databases|HTAP]], and now LTAP? I don’t think so.

A new term that popped up at [[Databricks Data & AI Summit 2026]] by Databricks announcement:

Zen Mode for Writing (Obsidian, Neovim)

Mon, 06 Jul 2026 11:53:35 +0000

Zen mode is a plugin most writing apps provide. I created my own in [[Neovim]] by hidding tmux bars and center text.

Music that Keeps You Focused

Mon, 06 Jul 2026 11:52:35 +0000

Many use music to stay focused when working or learning, but not every piece of music helps; some can lead to the opposite if it’s emotional, or if the lyrics are ones your brain starts listening to. Below are some sites and music providers that are mostly neuroscientifically tested to help you [[Deep Work|deeply focus]].

Tokenmaxxing

Mon, 06 Jul 2026 08:51:37 +0000

Came up with the [[Vibe Coding]] when you need to use all tokens as part of your subscription. Have a [[Vibe Code Agents]] running at all times. Something that came up on Twitter. Especially in combination with personal agents like [[OpenClaw]] and others.

Editing and Proofreading Your Writings

Mon, 06 Jul 2026 07:05:23 +0000

Adding more [[restrictions]] is a good thing.

Seeing [[Paul Graham]] only using [[vim#^a5d111]] and email for writing his blogs and getting feedback it got me thinking that all these Google Docs and easy-to-comment documents make it easier to give feedback, a comment that might not be well thought through. But giving more work to the author makes him more insecure.

Classical Architecture of Data Warehouse: Staging -> Cleansing -> Core -> Marts

Mon, 06 Jul 2026 07:01:59 +0000

This is the classical data warehouse architecture I learned at the beginning of my career. And to this day, I like to [[Data Modeling|model]] my data warehouse this way.

Death of the Follower: TikTokification

Sun, 05 Jul 2026 17:57:35 +0000

The death of the follower is more relevant than ever, even though the talk was given two years ago. Today, even LinkedIn and many other platforms are TikTok-ified.

Vibe Coding: Think Slow, Cultivate Taste, and Build Fast

Fri, 03 Jul 2026 08:23:40 +0000

Vibe coding is the next evolution of [[Prompt Engineering]], when you instruct your [[AI Agents]] to run autonomously. The best example is [[Claude Code]], but there are more.

Writing with AI: What's the Best Approach?

Fri, 03 Jul 2026 08:05:30 +0000

To me, AI writing always results in this:

They always look nice at first sight, but over time, they look all the same: complex, unnatural, and busy. Same as [[DALL-E]] images.

The Differences between Writing and Coding

Fri, 03 Jul 2026 07:54:35 +0000

[[AI Writing]] differs from coding because it’s consumed by different stakeholders. One is a computer running the code; the other is a real human being.

I'd rather Read the Prompt

Fri, 03 Jul 2026 07:40:49 +0000

A great take away form how I see the AI generated content, instead of seeing the summary of a persons prompts, I’d like to see the actual input and what the person was curious to know.

How to Detect AI Writing

Fri, 03 Jul 2026 07:18:45 +0000

Sometimes it isn’t as easy to spot. But there’s also the clear an obvious signs. Like below points, if a couple of them are combined, it’s very sure.

Data Modeling Languages: Query & Semantic Layer Evolution

Thu, 02 Jul 2026 15:51:33 +0000

Data modeling languages or query languages. Languages above [[SQL]].

Languages

What will be interesting, is the language we use for defining these metrics. It started with MDX, then DAX, then later came LookML and now we have Malloy and others:

Declarative vs. Imperative

Thu, 02 Jul 2026 15:45:06 +0000

The difference between [[declarative]] and [[imperative]] is the describing of how something is done. An imperative data pipeline, for example, needs to implement all the technical details of uploading to S3, implementing against the API.

DRY: Don't Repeat Yourself

Thu, 02 Jul 2026 15:22:27 +0000

“Don’t repeat yourself” (DRY) is a foundational principle in software development, focusing on reducing the repetition of software patterns, substituting them with abstractions, or employing data normalization to eliminate redundancy.

Query Databases in Neovim (DBUI)

Thu, 02 Jul 2026 10:53:16 +0000

Compared to other [[SQL IDEs]], I like using vim motions and [[Neovim]] for querying and copy the data in a new buffer. There’s two plugins that make this possibel:

Static Site Generators (SSG)

Wed, 01 Jul 2026 18:51:40 +0000

A static site generator (SSG) is a software used to generate static websites. Usually in plain [[HTML]] with added CSS, and potentially little JavaScript.

Open-Source Data Engineering Projects (2022-2026)

Wed, 01 Jul 2026 16:36:10 +0000

A curated list of many open-source data engineering projects collected over the years.

This note is for data engineers and developers who like to get their hands on some hands-on projects, hand-picked with care.

Future of Blogging: More Interconnected

Wed, 01 Jul 2026 14:43:55 +0000

I believe the future of blogging is more interconnected, with notes that are continuously updated and blog posts written as one-offs.

No Meetings: Why I Choose Async Over Short Calls

Wed, 01 Jul 2026 13:14:47 +0000

Why don’t I do meetings or short calls?

I favor async text-based communication: I can do it on my terms and when I have time.

Digital Garden

Wed, 01 Jul 2026 09:09:35 +0000

A digital garden (sometimes also called a [[Digital Oasis]]) breaks away from the traditional structure of a “personal blog” or a glossary, which is mainly linear in nature. Unlike these platforms, where content is presented in a series from an index page, a digital garden offers a multi-dimensional exploration. Here, you can delve into the garden, moving inward to discover various interlinked and sometimes unfinished notes, which I refer to as [[Continuous Notes]]. These are dynamic, continuously updated with fresh insights, representing a novel category that I believe is the [[Future of Blogging]]; see also the different [[Types of Content Online]].

My Note-Taking Path History and Evolution

Wed, 01 Jul 2026 09:07:16 +0000

My path so far with note-taking:

Forgetting everything
Taking scattered and very detailed notes on multiple devices, apps, and paper
Improving during my studies with OneNote, where notes related to work or study go into separate notebooks (no notes for personal notes yet).
Starting to create a personal notebook for travels, personal research, etc.
There is still a lot of confusion about:
1. Where to store my notes
2. Changing of the folder structure
3. Finding older notes is complex and rarely happens
Switching to [[Obsidian]] with a new open format and a different spirit and capabilities.
1. See [[Todays Daily Graphs - Obsidian Graph|my Obsidian Graph Evolution]] from 2021 to today.
Starting my [[Second Brain]] with [[Public Second Brain with Quartz|Quartz]]
1. Constantly updating my long-time wealth of personal knowledge by adding notes about my health, journals, cooking, books I read, and everything related to my life.
2. I started to connect notes and sophisticate my system in a way that I can confidently find it later in my life span, the moment I need it.
Start using [[Vim]] and, more importantly, its [[Vim Language (and Motions)|motions]] for fast and effortless note-taking.
Sharing them publicly with [[Quartz|Quartz]].

Origin: [[My Obsidian Note-Taking Workflow]] at My Obsidian Note-Taking Workflow | ssp.sh

Data Engineering Vault

Wed, 01 Jul 2026 09:02:03 +0000

Data Engineering Vault

Welcome to the Data Engineering Vault, an integral part of my larger [[_index|Second Brain]]. This curated network of data engineering knowledge is designed to facilitate exploration, discovery, and deep learning in the field of data engineering. Here, you'll find a rich ecosystem of 1000+ interconnected terms and [[Data Engineering Concepts|concepts]], each serving as a gateway to deeper insights. Functioning like a Digital Garden for data engineering, this network allows you to organically explore and connect ideas.

My Today's Daily Graphs - Obsidian Graph

Wed, 01 Jul 2026 07:44:48 +0000

This is my personal collection of Today’s Graph from my [[Obsidian]] and [[Second Brain]].

Graphs

See all graphs in chonological order.

Obsidian

Wed, 01 Jul 2026 07:43:22 +0000

Obsidian is a [[Second Brain]] and note-taking app. It is a powerful and extensible knowledge base that works on top of your local folder of plain text files.

Home Made Ramen: Step-by-Step Instructions

Tue, 30 Jun 2026 15:32:36 +0000

This is a delicous Ramen soup, that you can make at home. It takes some time, but it’s fully worth it.

Data Model Engine

Tue, 30 Jun 2026 15:21:13 +0000

A Data Model Engine is a system or framework that can model data. It is typically used in a [[Semantic Layer]] or as a layer between data and [[Data Visualization]]. Enable efficient querying, transformation, and analysis, often abstracting heterogeneous data sources and optimizing query execution for performance.

Ramen - Soy Sauce Eggs

Tue, 30 Jun 2026 14:55:48 +0000

Original Recipe

Ingredients

4 large free-range eggs
200 ml low-salt soy sauce
1 splash of vinegar (instead of mirin)
1 thumb-sized piece of ginger
2 cloves of garlic

Instructions

Boil the eggs for 5 minutes, then refresh in cold water and peel
![[Ramen - Soy Sauce Eggs-20240909112210542.webp]]

Data Engineering YouTube Channels

Tue, 30 Jun 2026 07:30:36 +0000

Popular YouTube Channels on [[Data Engineering]].

Seattle Data Guy by [[Ben Rogojan]]: First YT on the topic of date engineering.
Kahan Data Solutions by [[Michael Kahan]]: Amazing explanations for DE terms. Especially on the topic of [[Data Modeling]].
Mehdio DataTV by [[Mehdi Ouazza]]: Great production-quality DE videos.
Advancing Analytics by [[Simon Whiteley]] and Co: All about Databricks & Data Engineering, bridging the gap to Business intelligence.
Darshil Parmar Channel by [[Darshil Parmar]]: Weekly new content.
The Data Engineering Channel by [[Chris Gambill]]
Learn Data Engineering with [[Andreas Kretz]]
Data with Zach with [[Zach Wilson]]
Joe Reis Show by [[Joe Reis]]
DataTalksClub by [[Alexey Grigorev]]: Community for data scientists, ML engineers, analysts, and AI practitioners with free courses.
Ronald van Loon: Helps and advises businesses in implementing successful data and analytics strategies to reach and surpass their goals
The Full Data Stack Channel: Help data people widen their “T” shaped knowledge across the full data stack by [[Hoyt Emerson]].
StartDataEngineering by [[Joseph Machado]]
It’s About Data with [[Matt Housley]] and [[Tony Baer]] ^214161

See more on [[Data Engineering#Community and Learning]] and [[People of Data Engineering]].

Plain Text Files

Mon, 29 Jun 2026 12:31:10 +0000

Knowhow stored in Notions and Google Docs will not scale (what if you need to change something?), we can’t do a search and replace. But why not use what developers have for ages for knowledge, too?

Kanagawa

Mon, 29 Jun 2026 11:28:35 +0000

It’s my favorite Color Theme, it’s a mix between gruvbox and tokyonight!, announced on Reddit.

I even created my own Obsidian Theme.

Color Palette

(from GitHub - rebelot/kanagawa.nvim: NeoVim dark colorscheme inspired by the colors of the famous painting by Katsushika Hokusai.)

Git Diff Report (HTML, txt)

Sun, 28 Jun 2026 10:57:42 +0000

Export a [[git difftool]] as text, HTML, PDF, or image for seeing and sharing changes with others in a nice visual way.

TUIs

Sun, 28 Jun 2026 10:52:22 +0000

Modern application with a UI running in [[Terminals]].

Tools

Development & APIs

posting - [[API Client]] like [[Postman]]: A modern API client that runs in the terminal.
- Neovim option: kulala.nvim
sql-tap: Watch SQL traffic in real-time with a TUI

AI

[[OpenCode]]: AI prompter like claude code
See more Agents at [[Vibe Code Agents]] such as [[Amp Code]], and more.

DevOps & Infrastructure

[[lazydocker]]: Terminal UI for docker management and monitoring with easy container controls
[[CDF (Continuous Data Framework)]]: Craft end-to-end data pipelines and manage them continuously within the terminal, on top of SQLMesh and dlt. ^3ce141
[[k9s]]: Kubernetes CLI to manage and navigate clusters in your terminal
Flowrs: Flowrs is a TUI application for Apache Airflow that allows you to monitor, inspect and trigger Airflow DAGs from the comforts of your terminal. ^e67697

Database Management

[[Query databases in vim (vim-dadbod)|Dadbod]]: Database management and query execution tool for Vim/Neovim
[[Harlequin]]
Rainfrog: A database tool for the terminal.
see more on [[SQL IDEs#Terminal based]]

Data & Streaming

kplay: Inspect messages in a [[Apache Kafka|Kafka]] topic in a simple and deliberate manner

System & Linux

[[btop]]: Resource monitor that shows CPU, memory, processes, and network usage ([[CPU Monitoring (Resources) on Linux]])
impala: TUI for managing wifi on Linux. ^4e20f0
bluetui: TUI for managing bluetooth on Linux. Similar to impala for Wifi.
hyprmon: TUI monitor configuration tool for Hyprland with visual layout, drag-and-drop, and profile management.
caligula: A user-friendly, lightweight TUI for disk imaging and ISO burning.
pacseek: A terminal user interface for searching and installing Arch Linux packages

File & Storage

[[yazi]]: File manager like [[Ranger - Console File Manager]]. There is also lf
[[AWS S3|S3]] client TUI for Amazon: [[stu]] Example
- Better to use [[s3fs-fuse]]

Chat in the terminal with open source [[Zulip]]: Official Zulip terminal client ^0c6e64
[[Bluesky]] on [[Bluesky#TUIs]]
circumflex : It’s [[HackerNews]] in your terminal. Example
yt-browse: it’s like search, but for [[YouTube]]
itter.sh: Social Media for Purists.
- register: ssh register:USERNAME@app.itter.sh
- login: ssh USERNAME@app.itter.sh
Lazycut: A simple terminal UI for video trimming

Productivity & Finance

Bagels: Powerful expense tracker that lives in your terminal.
- Creates a config.yaml and uses [[SQLite DB|SQLite]]
Ticker: Track stocks, crypto, and derivatives prices and positions in real time from your terminal
calcure: Modern TUI calendar and task manager with minimal and customizable UI.
Gloomberb: open-source finance terminal

Web & Information, RSS

Newsboat: RSS Feeer in the terminal. More [[RSS Readers]]
wttr.in: The right way to check the weather
Browsh: Fully text-based browser.

Media & Downloads

Cliamp: Terminal Winamp
Surge: Surge is a blazing fast, open-source terminal (TUI) download manager built in Go.

Emails

[[Neomd]]: My own email client, a minimal terminal email client for people who write in Markdown and live in Neovim.
[[Pop Email]]: Modern CLI email client with a clean interface for sending emails from your terminal
[[Neomutt]] / [[Mutt]]: Feature-rich command line email client with powerful customization options
msgvault: Analytics on top of emails by [[Wes McKinney]] ^d6daa8
HEY CLI: TUI for [[HEY|Hey.com]], and Agent Skills.
aerc: a pretty good email client
matcha: A functional email client for your terminal, built with Go and the charming Bubble Tea TUI library. ^517a62

SSH GUIs

[[SSH]] with a TUI: lazyssh: A terminal-based SSH manager inspired by lazydocker and k9s - Written in go.

Git and GitHub

[[lazygit]]: Simple terminal UI for git commands, making version control more intuitive
[[Working with GitHub PR within Neovim]].
ghui: Terminal UI for browsing and acting on your open GitHub pull requests across repositories.
hunk: Review-first terminal diff viewer for agentic coders. Used by [[Mitchell Hashimoto]] ^1fe228
lumen: Beautiful git diff viewer, generate commits with AI, get summary of changes, all from the CLI.
oyui: Modern TUI merge tool and interactive diff editor for Jujutsu ([[jj (git)]]) and Git. ^1e19c9
See also [[How to Review a PR without Leaving the Terminal (Neovim)]]

Trains and Information

sbb-tui: TUI client for Switzerland’s public transports timetables, inspidered by the SBB/CFF/FFS app

Lists

awesome-tuis: List of projects that provide terminal user interfaces
My GitHub List

Example

![[TUIs-20240904150525832.webp]]

Quartz: Publish Obsidian Vault

Sun, 28 Jun 2026 08:46:29 +0000

Quartz is a fast, batteries-included static-site generator that transforms Markdown content into fully functional websites. Created by [[Jacky Zhao]]

Quartz is a free alternative to [[Obsidian]] Publish. Other once can be found in [[Open-Source Obsidian Publish Alternatives]].

Open-Source Obsidian Publish Alternatives

Sun, 28 Jun 2026 08:46:27 +0000

As Obsidian Publish is great but expensive, I’m interested in an open-source solution that I can integrate into my blog.

Here are some examples:

Other Public Second Brains

Sun, 28 Jun 2026 08:46:27 +0000

Besides my Second Brain (see how it’s built on [[Public Second Brain with Quartz]]).

Here are some other good ones:

Andy Matuschak
Strikingloo - Luciano Strika
David Gasquez
Brandon Boswell
jzhao.xyz (Founder of [[Quartz]])
Neocortex - Yigit Colakoglu
Eat Sleep Data
Fulcra - Ryan J. A. Murphy
Pelayo Arbués
Dr. Mario’s 🧠
TfT Hacker
The Integral Guide to Well-Being
Swyx’s Second Brain
mnml’s vault
Fork My Brain
Nikiv: My Knowledge Wiki 🌿
Lough on Data: Experiments, experiences, tips, and discoveries about data engineering.
Maggie Appleton ([[Maggie Appleton]])
Gilbert Sanchez Garden
Marc Bielert Obsidian Vault
DataVidhya Course Notes & Resources by [[Darshil Parmar]]
Joseph Jude
Digital Garden by [[Colophon]]
Matt Broberg
Welcome to Socratica
TopicPartition by [[Stanislav Kozlovski]]
Ewans Florilegium Quartz Brain (His tour at Interview with Obsidian enthusiast 2026 - YouTube)
Joschua Quartz Brain
Jochens Second Brain

Collective lists:

Public Second Brain with Quartz

Sun, 28 Jun 2026 08:46:27 +0000

This is my documentation of how I publish my notes from a private [[Obsidian]] vault to my Public Second Brain (this very site) with open-source [[Quartz|Quartz]] and [[GoHugo]].

Convert Markdown to Rich Text (Google Docs)

Sun, 28 Jun 2026 08:46:26 +0000

When you work with [[Markdown]] you inevitably will need an efficient way to convert from Markdown to [[Rich Text]] sometimes, e.g., [[Google Docs]]. Tweet

Create a Book on Markdown

Sun, 28 Jun 2026 08:46:26 +0000

For creating a Book on top of [[Markdown]], there are some solutions I want to explore here.

This could be done solely in [[Obsidian]], but there wouldn’t be management around chapters, post-processing, and many other things that include a book.

My Tech Stack

Sun, 28 Jun 2026 08:46:26 +0000

This is what I use behind the scenes of [[My Websites]], ssp.sh.

Web

Blog: [[GoHugo]] with a theme called “Blogger”, customized colors to [[Kanagawa]].
- Theme: LoveIt - A clean, elegant but advanced blog theme for Hugo.
- See also [[My Websites Color Palette]].
Second Brain and DE Vault: [[Quartz]].
- See the code here and how I [[Public Second Brain with Quartz|publish my Second Brain with Quartz]].
- See also the My Obsidian Note-Taking Workflow with the YT video for more details.
DEDP Book: I use mdBook for creating my book. See more on [[mdBook]].
Newsletter List: I use [[Listmonk]] hosted on [[Railway.app]].
- See details [[Listmonk#Sending Emails of Newsletter]] and how I send emails.
Analytics with [[GoatCounter]], self hosted on [[Railway.app]] (with [[SQLite DB|SQLite]] DB :)

Everything is based on Markdown

Everything is based on [[Markdown]]. This is the key to me publishing as much as I do, as the publication process involves no friction and, even more importantly, no formatting. See also more in My Obsidian Note-Taking Workflow (YT).

Should I Change My Writing Style to Short Stories, because of AI and Low Attention Span?

Sat, 27 Jun 2026 17:29:38 +0000

[[Short-form writing]] just got much more popular, with the [[Attention Span declining|low attention span]] of people with [[TikTok]] and YT shorts. The movies get more repetitive as the audience can’t follow the movies anymore, or studies say that the average attention span is something like 40 seconds, to write short, repetitive articles, and so does potentially change writing.

Second Brain

Fri, 26 Jun 2026 16:10:16 +0000

In simple terms, it’s just note-taking in a single app, but instead of organizing your notes into folders, you link them. It’s what [[Obsidian]] made popular. The name, because I use it for all facets of life, if I go to the doctor or dentist, I write it down, if I write an idea, or journal, it’s all there. [[Tiago Forte]] coined the term initially.

Best Writer Pencil

Thu, 25 Jun 2026 10:43:09 +0000

For me it’s the FriXion Clicker, easy to remove, very great flow on plain [[Paper Notebooks|Physical Notebooks]], and cheap. Easy refill.

Physical/Paper Notebooks

Thu, 25 Jun 2026 10:41:45 +0000

buying expensive notebooks will male you write more.. Reddit

I am not sure if I agree with the expensiveness of notebooks, but I found one that I really like (see below [[#My Favorite Noteboook, and Best Physical Notebooks for me]] for more); the pen really flows nicely, the paper feels quality, and the size and whiteness are delightful. I bought five.

SQL Query Engine

Thu, 25 Jun 2026 08:53:31 +0000

Products:

[[Presto]]
[[Trino]]
[[Spark SQL]]
[[Dremio]]
[[Photon Engine]]
- as of [[2026-06-16]], added [[Reyden]]
[[Apache Hive]] (why?)
[[DuckDB]]
[[Cloud Data Warehouses]] like snoflake, biquery etc.
[[dbt Fusion]] (acquired [[SDF]])
Husky by [[DataDog]]
[[Quanton]] by [[Onehouse]]
See more in [[Apache Spark Alternatives]]

See more on [[Emerging Architectures for Modern Data Infrastructure (a16z)]], SQL Reference - FoundationDB Record Layer documentation.

Federated Query Engines

Thu, 25 Jun 2026 08:36:13 +0000

Common [[Data Virtualization|Data Federation and Virtualization]] tools that are used out there.

SQL Query Engines

Trino, DuckDB, etc. find the full overview at [[SQL Query Engine]].

Note-Taking: Digital vs. Paper

Thu, 25 Jun 2026 08:15:13 +0000

Writing is therapy! I usually write as much as possible on my laptop because I can type as fast as I think. At the same time, my handwriting can’t keep up with my thinking speed. There is no slowing down and forgetting ideas or thoughts. On the other hand, I can reformat, re-arrange, add, and delete, which will help my thinking process which wouldn’t happen in my brain. The advantage of pen and paper is that I use different muscles and brain activities when I write, which helps me think differently. I usually use them when I need to outline my blog post, if I’m stuck or distracted on something, or if I go out in nature and only bring my physical journal.

DuckCon #7 - Amsterdam (Event Notes)

Thu, 25 Jun 2026 06:32:07 +0000

Conference about [[DuckDB]].

![[img_DuckCon 7 - AMS_1782306134718.webp]]
When: [[2026-06-24]]

DuckCon #7 in Amsterdam – DuckDB
YouTube: DuckDB Live Stream

Notes and Slides

Find full presentation titles and speakers in [[#Agenda|Agenda]] header at the bottom.

Today's Office: A Visual Log

Wed, 24 Jun 2026 14:36:25 +0000

Today’s office is something I am doing for a long, long time, it’s taking a short snap of today’s [[Computer Desk Setup & Monitor Workflow|computer setup]], and post it on social with #todaysoffice hashtag.

My Distraction-Free Typewriter (Micro Journal)

Wed, 24 Jun 2026 11:52:49 +0000

I finally got my baby, a [[Distract-Free Typewriter]]. I chose the [[Micro Journal#Rev 2|Micro Journal Rev.2]]. Thank you, [[Un Kyu Lee]]. As a writer and author, this thing is a beauty and just what I didn’t know I wanted.

Computer Desk Setup, Monitor & Developer Workflow

Wed, 24 Jun 2026 11:52:15 +0000

My desk setups progressed over time. From side-by-side displays to an ultrawide and portrait combo, and finally to a single 27’’ or an extended laptop setup for meetings.

Pollmd: My Minimal Survey Service (for Newsletter) that Works with Markdown

Wed, 24 Jun 2026 09:53:07 +0000

Pollmd is my minimal (newsletter) survey with dynamic questions right from Markdown (links) with one time installation with [[Quack Protocol (DuckDB)|DuckDB Quack Protocol]].

'Coffee Break Rhythm': Moving around with Redundant Offices

Wed, 24 Jun 2026 09:50:43 +0000

As learned in Redundant Office podcast with YouTuber CGP grey, where he says that he is doing a burst of work while walking around in London. Something I found that works for me, too.

Dagster

Tue, 23 Jun 2026 15:03:06 +0000

Dagster is one of many [[Data Orchestrators]]. It operates in a [[declarative]] and [[data-aware]] manner, offering unique capabilities in data orchestration.

Business Intelligence

Tue, 23 Jun 2026 13:21:24 +0000

[[The Goal of Business Intelligence]] and the [[Problems of Business Intelligence]].
Close connected is [[Data Engineering]], which begs the question of [[Business Intelligence Engineer vs Data Engineer]].

Third normal form (3NF)

Mon, 22 Jun 2026 13:54:08 +0000

Third normal form (3NF) is a database schema design approach for relational databases that uses normalizing principles to reduce the duplication of data, avoid data anomalies, ensure referential integrity, and simplify data management.

DuckLake by DuckLabs

Mon, 22 Jun 2026 11:28:54 +0000

DuckLake is a [[DuckLabs]] approach to simplifying [[Data Lakehouse|Lakehouse]] architectures using a standard SQL database ([[DuckDB]]) for all metadata instead of complex file-based systems while still storing data in [[Data Lake File Formats|Open File Formats]] like [[Apache Parquet|Parquet]].

How to Get Started with Data Engineering

Mon, 22 Jun 2026 09:51:01 +0000

You might work in a different field or be curious about what [[Data Engineering]] is, then this is the right place to get started as a data engineer.

What is Data Engineering

Mon, 22 Jun 2026 09:36:44 +0000

Data engineering is the less famous sibling of data science. Data science is growing like no tomorrow, as is data engineering, but it is much less heard of. Compared to existing roles, it would be a software engineering plus business intelligence engineer, including big data capabilities across the [[Hadoop]] ecosystem, streaming, and computation at scale.

Writing Manually (In Times of AI-generated Content)

Mon, 22 Jun 2026 06:38:01 +0000

In times of [[AI Writing]], writing manually is more important than ever. As with many [[Generative AI|AI Generated]] texts, I’d rather see the prompts, it would have more soul and character than the output.

Smart Note Taking

Sat, 20 Jun 2026 20:24:27 +0000

Why is this important? Because without the smart, you’ll add more clutter to your system, leading to minor organization and more stress.

AT Protocol

Fri, 19 Jun 2026 08:56:32 +0000

The AT Protocol at:// (Authenticated Transfer Protocol, or atproto) is a standard for public conversation and an open-source framework for building social apps.

Data Engineering: Trends and Predictions (2022-2026) 🔮

Fri, 19 Jun 2026 06:21:54 +0000

Here I’ll list my predictions and trends, mainly for myself and to keep a record of what’s happened in the past.

Routines such as Sports or Meditation

Wed, 17 Jun 2026 21:43:15 +0000

Many studies say that we should start a [[Routine]] or it doesn’t stick. I’m not too good at this, but I try to do it primarily for my weekly sports, [[Meditation]], or writing habits. But as with kids, it is not as easy as before. I do my best whenever possible and do not try to feel bad if I can’t find the time. As it’s important to me, as any parent knows, there is no more important task in life than being a parent and spending undistracted time with your kids.

Being a Morning Person

Wed, 17 Jun 2026 21:41:55 +0000

Loved this small essay on On becoming a day person. It is something I had adapted with kids as well. I don’t have the early morning [[Routine]], but I agree with almost everything [[Herman Martinus]]’s says.

The Ordinary Life: And how a Good Life Might Look Boring

Wed, 17 Jun 2026 21:20:57 +0000

Doing the same every day, like a life of a novelist or [[writers]] (inspired also by the book [[Novelist as a Vocation (Haruki Murakami)]]) is to be called a boring life.

Vim Motions for Writers

Wed, 17 Jun 2026 20:25:36 +0000

[[Vim Language (and Motions)|Vim Motions]] for [[writing]] are the best [[Deep Work|flow]] I can imagine. The smoothness and clarity are on another level.

Neovim

Wed, 17 Jun 2026 20:06:57 +0000

Text editor evolved from the old [[vim]]. People call it not [[IDE]], such as [[VSCode]] or similar, but [[PDE (Personalized Development Environment)]].

Quack Protocol: Client-Server Architecture for DuckDB

Wed, 17 Jun 2026 11:16:35 +0000

Allowing [[Client-Server Architecture]] over the network (TCP/IP with HTTP) for [[DuckDB]], opening up new [[DuckDB Use Cases]].

Quack, a protocol that turns DuckDB into a client-server database. True to DuckDB’s philosophy, Quack is simple and fast. You can set it up in seconds and it delivers high-performance remote access, turning DuckDB into a full-fledged general-purpose database system.

Uni-taskers, and Single-use Devices

Wed, 17 Jun 2026 09:54:47 +0000

Devices such as [[My Typewriter (Hermes 2000)]], not multi-taskers, but devices that doe one thing best, and one thing only.

Local First

Wed, 17 Jun 2026 09:42:13 +0000

An outstanding essay is on local first and how to resolve the best of Markdown with a local plain text file with a method called [[CRDT]]. As the article mentions, Towards a better future. As other technologies like [[Google Docs]], Notion, etc. always lack something, CRDTs seem to solve all 7 points and satisfy the local-first ideas.

Vimrc for Obsidian

Wed, 17 Jun 2026 09:40:40 +0000

These is my [[vim]], or more specific, [[Vim Language (and Motions)|Vim Motions]] setup for [[Obsidian]].

Set commands in `.vimrc`

And then you can set a custom shortcut with specifying the name first and nmap it to a shortcut like this:

Add Blog/Book to my Obsidian Vault via Symlink

Wed, 17 Jun 2026 09:40:22 +0000

This is a note about how I link my book and Blog to my Obsidian Second Brain Vault. Both of the folders are git repos that usually I edit in a separate Vault or with Neovim. But with Symlinks, thanks to Lazar I linked them directly to my main Vault and I can edit them now from within one single Vault.

Pathless Path

Wed, 17 Jun 2026 08:45:38 +0000

The [[Pathless Path]] is experimenting on your own to find a life you want to live vs. “getting ahead” on a script others follow – embracing uncertainty & discomfort. Whereas the opposite, the [[Default Path]] follows the planning/conforming path.

Focus@Will

Wed, 17 Jun 2026 08:05:21 +0000

Non-distractive music for work.

I have a lifetime subscription, as I use it all the time. It’s very effective. I blogged about it already on Do you listen to music while working? and Tools I Use – Part III.

DataFusion

Tue, 16 Jun 2026 18:08:11 +0000

DataFusion is an extensible [[Data Model Engine|query execution framework]], written in Rust, that uses [[Apache Arrow]] as its in-memory format.

DataFusion supports both an SQL and a DataFrame API for building logical query plans as well as a query optimizer and execution engine capable of parallel execution against partitioned data sources (CSV and Parquet) using threads.

Earn Attention with Writing

Tue, 16 Jun 2026 09:20:39 +0000

Write the lessons learned. The mesmerizing content is the content shared, whether there is a return or not.

You can either buy someone’s attention, or you can earn it. If you earn it, it’s vastly deeper. You put good stuff out to the cosmos; usually, it will return good stuff. [[The more you share the more you get]]

Out-Teach Your Competition

Tue, 16 Jun 2026 09:20:39 +0000

Sparked [[Earn Attention with Writing]] .

Notes

we don’t share when others don’t have something to learn from
- It’s vastly different if you share something people learn, or if you put out because you need to.
- you have two options: either
- David shares that the philosophy of out-[[teaching]] the competition is in their DNA.
most bigger unicorn companies, their approaches don;t apply to many companies.
- But what Basecamp shares is reaching many more as there are just more companies that can relay to what they are saying — basically in the same shoes
basically, they love sharing it even though it’s not perfect. it’s [[Learn in Public]]
they don’t like content marketing. Because they have to push stuff on a schedule. Basecamp doesn’t share if they don’t have anything to say
- also, sometimes people listen to [[David Heinemeier Hansson (DHH)]] and [[Jason Fried]] and sign up 10 years later. How can you justify this? That applies a lot to how we write ([[Writing]]) at [[Airbyte]].
- Why you won’t normally find the ‘good stuff’ in content marketing.
You can either buy someone’s attention or you can earn it. [[David Heinemeier Hansson (DHH)]]
- Now, on the bigger point though, this is actually good for business in the sense that you can either buy someone’s attention or you can earn it.
- … it’s just so much deeper.
- you put good stuff into the universe
- it’s a source of energy
  - And I would do it whether there was a return or not a return.
- old adage that if you really wanna learn something, you should try teaching it
- [[Sharing is caring]]

Trust

See [[Trust]].

On Writing.

Tue, 16 Jun 2026 09:20:38 +0000

Writing has been my new job after writing for myself for nine years. So, how do you get better, and what about writing makes it so hard?

Travel Locally, Where You Are

Tue, 16 Jun 2026 08:53:27 +0000

Many people dream of traveling to faraway places to experience new [[culture|cultures]] and sightseeing. Yes, you learn a ton, you learn new places, and the culture is hard to learn when not traveling.

How to Automate Newsletter Based on SecondBrain and Bluesky Changes

Tue, 16 Jun 2026 08:39:07 +0000

I automate the gathering of content for my [[Newsletter]], but not the writing of it. Everything mechanical that I used to do by hand — pulling links to recently-updated [[Second Brain]] notes, finding books I’ve read, copying top [[Bluesky]] posts — runs from a small Python script. The personal writing stays mine.

AI Orchestrators

Tue, 16 Jun 2026 07:36:41 +0000

Instead of [[Data Orchestrators|Orchestration]] for data engineering jobs, AI orchestration is all about scheduling and orchestrationg [[AI Agents]].

I’d say the easiest is to use [[tmux]] to schedule multiple Claude Codes agents. And you could also extend to run multiple branches of the same repo with [[git worktree]]s.

Querying Bluesky with DuckDB and SQL

Mon, 15 Jun 2026 21:17:37 +0000

Querying [[Bluesky]] directly with [[DuckDB]] and [[SQL]] is easily possible because the [[AT Protocol]] is publicly available and has a [[Jetstream]] that you can read.

Data Modeling

Mon, 15 Jun 2026 11:31:31 +0000

Data modeling has changed; when I started (~20 years ago), choosing between [[Bill Inmon|Inmon]] and [[Ralph Kimball|Kimball]] was common.

Today, in data engineering, data modeling creates a structured representation of your organization’s data. Often illustrated visually, this representation helps understand the relationships, constraints, and patterns within the data and serves as a blueprint for gaining business value in designing data systems, such as data warehouses, lakes, or any analytics solution.

The Role of a Data Engineer

Mon, 15 Jun 2026 10:39:51 +0000

In order to get high-quality and frequently updated data sets, it is important to distinguish between data pipelines that are done and cleaned by data engineers and all the others that are mostly exploratory. We at Airbus use a folder that is called “cleaned” and all data sets produced there are constantly updated, documented, and of the highest quality. Based on these data sets you create your own. We use the data lake solution [[Palantir Foundry]] (brand name of Airbus: Skywise) which provides you with a map where you see the data lineage easily. Documentation and metadata to each data set are crucial as otherwise, you lose the overview of your data, which is also one main task of a data engineer.

Data Analytics

Mon, 15 Jun 2026 10:38:05 +0000

The term data analytics comes mostly from the success of [[dbt]]. As it was so popular, and allowed to join together SQL statements in an elegant and documented way, the field got a new name, “Data Analytics”.

Vibe Code Agents

Mon, 15 Jun 2026 10:31:31 +0000

[[Vibe Coding]] is a new phenoment, and these are some of the tools for it. These [[AI Agents]] run mostly autonomously in the [[Terminals|Terminal]].

Data Transformation

Fri, 12 Jun 2026 12:30:22 +0000

Data transformation is the process of converting data from one format into a different format. Reasons for doing this could be to optimize the data for a different use case than it was originally intended for, or to meet the requirements for storing data in a different system. Data transformation may involve steps such as cleansing, normalizing, structuring, validation, sorting, joining, or enriching data.

Data Literacy

Fri, 12 Jun 2026 12:24:09 +0000

Data literacy is the ability to derive meaningful information from data, just as literacy in general is the ability to derive information from the written word.

Rollup

Fri, 12 Jun 2026 12:22:31 +0000

Rollup is a form of summarization or pre-aggregation. Rolling up data can dramatically reduce the size of data to be stored and reduce row counts by potential orders of magnitude.

Apache Spark

Fri, 12 Jun 2026 12:20:11 +0000

Apache Spark™ is a multi-language engine ([[Cluster-computing frameworks]]) for executing [[Data Engineering]], [[Data Science]], and [[Machine Learning]] on single-node machines or clusters.

Lambda Architecture

Fri, 12 Jun 2026 12:19:24 +0000

Lambda architecture is a [[big data]] processing pattern designed to handle massive datasets by combining batch and real-time (stream) processing.

Data Swamp

Fri, 12 Jun 2026 12:17:50 +0000

Data swamps start to arise when there is a lack of responsibilities, data ownership, availability, and [[Data Governance]]. It’s when a [[Data Lake]] is unmanaged or unable to provide value. Sometimes a Data Swamp can also arise from a [[Data Warehouse]] due to existing hybrid models.

Apache Hudi

Fri, 12 Jun 2026 12:17:36 +0000

Hudi is a rich platform to build streaming [[data lake]]’s with incremental [[data pipelines]]
on a self-managing database layer, while being optimized for lake engines and regular [[batch processing]].

Apache Hive

Fri, 12 Jun 2026 12:17:20 +0000

Apache Hive is a data warehouse software project built on top of Apache Hadoop for providing data queries and analysis. [2] Hive gives an SQL-like interface to query data stored in various databases and file systems that integrate with Hadoop. Traditional SQL queries must be implemented in the MapReduce Java API to execute SQL applications and queries over distributed data. Hive provides the necessary SQL abstraction to integrate SQL-like queries ( HiveQL) into the underlying Java without the need to implement queries in the low-level Java API.

April Newsletter 2023

Fri, 12 Jun 2026 12:15:24 +0000

Hello friends!

Welcome to the new format of my newsletter! I aim to create a more informal space for updates and discussions on Data Engineering and other topics I explore as I create new content. Expect to receive not just updates on new blog posts, but also interesting tidbits from my life and work.

Open Standards

Fri, 12 Jun 2026 12:14:36 +0000

With the explosion of tools in the open-source data ecosystem, there is the need to couple them, maximize the tools into a single data stack, and simplify along the way. To achieve this, we need open foundations that everyone is implementing on.

Jinja Template

Fri, 12 Jun 2026 12:13:40 +0000

Jinja stands out as a fast, expressive, and expandable templating engine. It incorporates special placeholders in the template, allowing for code writing in a manner akin to [[Python]] syntax. Subsequently, this template is supplied with data to produce the final document.

SELECT Insights - Bundling with Microsoft Fabric and Orchestration (#2)

Fri, 12 Jun 2026 12:13:18 +0000

Hello friends,

A quick note before we start: I changed the name of this newsletter to “SELECT Insights” and changed the structure slightly. I plan to follow this structure for the upcoming editions. If you want to know more, I have more details on newsletter.ssp.sh. — And on another exciting side note, I also move the domain from sspaeti.com to ssp.sh. I hope you like it; more on the history in this Tweet. But now, let’s get into this month’s newsletter.

Data Lake Table Formats (Open Table Formats)

Fri, 12 Jun 2026 12:13:15 +0000

Prominent table formats include [[Delta Lake]], [[Apache Iceberg]], and [[Apache Hudi]].

Data lake table formats serve as databases-like features on top of distributed [[Data Lake File Formats|File Formats]]. Similar to a traditional table, these formats consolidate distributed files into a singular table, simplifying management. Consider them an abstraction layer that structures your physical data files into coherent tables.

Convert Smartphone to Dumb Phone

Thu, 11 Jun 2026 21:35:35 +0000

Inspired by YouTube how to convert to a dumb phone, I tried to convert my “smart” phone to a “dumb” phone, to use it more intentionally. To avoid doing everything with willpower, which is quite hard if all the tech is engineered to make you [[Social Media Addiction|addicted]].

Turn Off, Shut Down, Re-energize

Thu, 11 Jun 2026 20:19:15 +0000

Something that is easy to say, but more important to do. I enjoy walks in [[nature]] or go for a long bicycle rides. It does not really matter what you do, but that you do something, and without distraction.

What I Have Learned Abroad

Thu, 11 Jun 2026 19:57:18 +0000

Maybe you’re thinking of starting one or two years abroad or traveling the world as well. Therefore, I want to share what I learned from my recent experience.

Laptop/Notebook & Desktops for Linux (Hardware)

Thu, 11 Jun 2026 08:12:10 +0000

Hardware, desktops or notebooks optimized for [[Linux]].

Manufacurer

[[Framework Laptops]]
Good Linux Hardware · Omacom · DHH suggestions by [[David Heinemeier Hansson (DHH)|DHH]]:
- Framework 13: This is the best Linux laptop I’ve tried so far. It’s been my go-to for over a year now. It’s got a superb keyboard with delicious travel, a 3:2-aspect, matte, 258ppi screen that’s ideal for code, it’s fully user repairable, and the 61kwh battery gets you about 6 hours in mixed use. Highly recommended in both the high-end HX370 version and even the last-gen 7840U. (If the 6 hours of battery sounds a bit light to you, get a 74wh Anker powerbank to double that).
- Beelink SER8: This miniPC blew my mind with its power-for-dollar ratio. It’s less than $500, comes with 32GB RAM + 1TB NVMe, and an excellent AMD 8745HS chip.
- Beelink SER9: Nearly double the price of the SER8 at ~$929, but the HX370 adds ~20% higher performance (and much better bluetooth/wifi range). Not quite as good value as the SER8, but a superb machine for less than a thousand bucks none the less.
- Framework Desktop: This tiny little beast packs the insane AMD 395+ MAX chip, which with up to 128GB of shared memory, 16 CPU cores, and 40 GPU cores, is just a mad power house of a computer. And yet it’s only 4.5L and completely silent. You’ll pay for the privilege at around $2,000, but if you just want the best, this is it!
- Beelink EQR5: The ultra-budget option is the $299 AMD 5825U 8-core Zen3 powered EQR5. It’s a two-generation old chip at this point, but comes with 32GB RAM + 500GB NVMe. Surprisingly capable!
TUXEDO Computers - (The Tongfang is the same as tuxedo’s, but cheaper) / [[Tuxedo Laptops]]
[[System76]]
ROG Zephyrus G14 - [[David Gasquez]]
Buy a Linux laptop? Laptops with Ubuntu or Linux Mint | Laptopwithlinux
Reddit user choosing ThinkPad
New Intel Snapdragon X chip
- Dell XPS 13, Snapdragon X Elite with Snapdragon X chip, OLED display
- Lenovo with A Developer’s Review of a Snapdragon X Laptop (Lenovo Yoga Slim 7x) - Wesley Moore

Check Best Laptop Deals for recommendations.

Writing is the Thinking

Wed, 10 Jun 2026 13:06:58 +0000

Remember: [[Writing]] is the thinking. The more you write, the more notes you take, and the better of a thinker you become. You’ll read better with better questions in your mind. Without writing, there is no thinking, no progress.

Markdown

Wed, 10 Jun 2026 08:59:24 +0000

A declarative way of writing text.

As written by [[Derek Sivers|Derek]] Write plain text files and summarized them in my
[[Personal Knowledge Management Workflow for a Deeper Life]].

Delta Lake

Tue, 09 Jun 2026 15:19:48 +0000

Delta Lake is an open source project that enables building a Lakehouse architecture on top of data lakes. Delta Lake provides ACID transactions, scalable metadata handling, and unifies streaming and batch data processing on top of existing data lakes, such as S3, [[Azure Data Lake Storage (ADLS)|ADLS]], GCS, and HDFS.

Apache Iceberg

Tue, 09 Jun 2026 15:02:56 +0000

Iceberg is a high-performance [[Data Lake Table Format]] for huge analytic tables. Iceberg brings the reliability and simplicity of [[SQL]] tables to big data while making it possible for engines to safely work with the same tables, at the same time.

Normalization

Tue, 09 Jun 2026 07:51:37 +0000

Database normalization is the process of structuring a [[Relational Model|relational database]] in accordance with a series of so-called normal forms in order to reduce data redundancy and improve data integrity. It was first proposed in the early 1970s by British computer scientist [[Edgar F. Codd]] as part of his [[Relational Model]].

Denormalization

Tue, 09 Jun 2026 07:51:34 +0000

In the realm of [[Relational Databases]], denormalization is a strategic process where we introduce precomputed, redundant data into an otherwise normalized database structure. This technique is primarily employed to enhance the read performance of the database.

Keep AI Out of Your (Obsidian) Vault

Tue, 09 Jun 2026 04:53:09 +0000

Everyone is using [[Obsidian]] for [[Generative AI|AI]], or wants to use it to become more [[Productivity|productive]]. But I think it’s a dead end.

ETL (Extract, Transform, Load)

Mon, 08 Jun 2026 12:00:16 +0000

ETL, which stands for Extract, Transform, and Load, involves a three-phase process for moving data. Unlike [[ELT]], ETL requires pre-defining the data schema and completing transformations before the data reaches its final destination. In contrast, ELT involves loading data first and then transforming it.

Cloud Data Warehouses

Mon, 08 Jun 2026 07:41:47 +0000

Another approach is that you change your on-premise Data Warehouse to a Cloud Data Warehouse to get more scalability, more speed, and better availability. This option is best suited for you if you do not necessarily need the fastest response time and you do not have tera- or petabytes of data. The idea is to speed up your DWH and skip the layer of cubes. This way you save much time in the development, processing, and maintenance of [[OLAP|OLAP Cubes]]. On the other hand, you lose in query latency while you create your dashboards. If you mainly have reports anyway, which can run beforehand, then this is perfect for you.

Cloud Data Provider

Mon, 08 Jun 2026 07:41:39 +0000

Cloud provider for data-related services.

Provider

[[AWS]]
[[Google Cloud]]
[[Azure]]
[[Databricks]]
[[Hetzner]]
[[cloudflare]]
[[DigitalOcean]]
[[fly.io]]
[[Firebase]]
[[Vultr]]
![[stackoverflow-dev-survey-2024-technology-worked-with-vs-want-to-work-with-platform-worked-want-social.png]]

![[Big Cloud Vendors-20241018092914585.webp]]
Source StackOverflow Survey

HackMD: Collaborative Markdown

Sun, 07 Jun 2026 12:38:06 +0000

HackMD is [[Google Docs]] for Markdown. Effective team collaboration.

[[Markdown]] is awesome, the only disadvantage is it’s not collaborative. HackMD seems to solve this.

Data Integration

Sun, 07 Jun 2026 08:12:00 +0000

Data integration is a crucial process in the world of data management, where we bring together data from varied source systems to create a cohesive, unified view. This integration can happen in multiple ways, such as through manual effort, data virtualization, application integration, or by migrating data from numerous sources into a singular, integrated destination. We delve into these methods of data integration in the discussion below.

Data Integration CLI tools

Sun, 07 Jun 2026 08:12:00 +0000

[[Data Integration]] that supports CLI use cases.

Tools

[[dlt]]: Python-based data integration library designed from the ground up to improve upon Airbyte/Meltano. Perfect fit as a library rather than platform, with well-thought-through architecture and comprehensive connector ecosystem
[[Sling]]: Similar to dlt, recommended by Dagster team. Works well integrated with Dagster orchestrator, supports incremental loading with custom SQL
[[Steampipe]]: SQL-based, very lean tool but lacks advanced features like incremental loading, full load functionality, and UI. Lower customization for non-existing connectors compared to dlt
[[ConnectorX]]: Fast connector for data transfer
[[ingestr]]: Simple one-to-one copy tool. For more complex scenarios, dlt is recommended as better alternative
[[SDF]] (now [[dbt Fusion]])
[[Meltano]] (bought, see [[Data Engineering Acquisitions#^61399b]])

Further Reads

[[Data Integration Tools- Choose the Right One]]
[[Data Integration]]

Origin: [[Data Integration Tools- Choose the Right One]]

Run DuckDB in your Website

Fri, 05 Jun 2026 14:21:20 +0000

With a simple HTML script (there is also [[React.js]] and [[Vue]] example) - you can embedd a [[DuckDB]] with [[WebAssembly (WASM)|WASM]] query with like this:

Stream Processing

Fri, 05 Jun 2026 08:37:11 +0000

Opposite to [[batch processing]], where we process data in events.

Tools

[[Spark Structured Streaming]]

[[Apache Flink]] (The main difference to Spark is that Flink was built from the ground up as a streaming product. Spark added Streaming onto their product later)

[[Apache Kafka]]
[[Apache Samza]]
[[Bytewax]]: Not active anymore, see State of the project?

[[NATS]]
[[Fluvio]]

Streaming != Stream Processing

Great write-up by [[Stanislav Kozlovski]] at Event Streaming is Topping Out. See more at [[Streaming#^01d13a]].

SQLMesh

Thu, 04 Jun 2026 14:46:18 +0000

Don’t hack custom scripts or use half-baked tools. SQLMesh ensures accurate and efficient data pipelines with the most complete DataOps solution for transformation, testing, and collaboration

Data Engineering Concepts

Thu, 04 Jun 2026 08:39:22 +0000

If you want to be a data engineer, learn these concepts. The table below is a Map of Content (MOC) with 101 (the 101 on data engineering 😉) entry-point ideas that, between them, link out to the rest of the [[Data Engineering|Data Engineering Vault]] via backlinks. Read it top-to-bottom as a learning path, or jump in where you need depth.

Data Engineering Glossaries and Handbooks

Thu, 04 Jun 2026 08:28:08 +0000

Below you find a wealth of knowledge in the form of a Glossary or a Handbook.

Glossaries

This is a curated list of Data Engineering glossaries:

dbt Fusion Engine

Wed, 03 Jun 2026 23:03:46 +0000

The dbt Fusion [[SQL Query Engine|Engine]] shares the same familiar framework for authoring data transformations as dbt Core, while enabling data developers to work faster and deploy transformation workloads more efficiently. GitHub

Email

Wed, 03 Jun 2026 22:19:24 +0000

A big part of everyone is email. Although I argue against email and instead propose a way we (should) communicate, it’s still a big part of work or other areas of life. Unfortunately, it’s like an empty to-do list for everyone else to add tasks, making it dangerous. The GTD method from above integrates email into the workflow by going through your email in the weekly reviews and adding essential emails to your to-do list. I do that by forwarding the email to Nirvana which adds them automatically to my inbox.

Data Warehouse

Wed, 03 Jun 2026 18:38:20 +0000

A Data Warehouse (DWH), also known as an Enterprise Data Warehouse (EDW), represents the traditional approach to data collection, a practice established over 30 years ago. The DWH is crucial for integrating data from numerous sources, serving as a single source of truth, and managing data through processes such as cleaning, historical tracking, and data consolidation. It facilitates enhanced executive insight into corporate performance through management dashboards, reports, or ad-hoc analyses.

The Data Warehouse Toolkit by Ralph Kimball

Wed, 03 Jun 2026 18:37:52 +0000

This is the book that got me started with the [[fundamentals of data engineering]]. before [[Joe Reis]] wrote [[Fundamentals of Data Engineering (Joe Reis, Matt Housley)]].

BI Tools

Wed, 03 Jun 2026 14:44:09 +0000

Tools of [[Business Intelligence]], there are many.

Tools

Here a none complete list:

[[Apache Superset]]: Open-source data exploration platform born at Airbnb, connects to any SQL speaking engine with 40+ visualization types and petabyte-scale query capabilities.
[[Power BI|PowerBI]]: Microsoft’s BI platform with deepest Excel/Office 365 integration, DAX formula language, and Copilot AI baked into the broader Microsoft Fabric stack.
[[Tableau]]: The visual analytics pioneer built on patented VizQL technology that turns drag-and-drop actions into database queries, now part of Salesforce.
[[TARGIT]]: Danish BI suite (Decision Suite) focused on industry-specific, out-of-the-box analytics for verticals like automotive dealerships, heavy equipment, and airports.
[[Metabase]]: The “5-minute setup” open-source BI beloved by startups, pairing a no-code question builder with a SQL editor and friendly UX.
[[Kibana]]: Visualization front-end for Elasticsearch, purpose-built for log, metric, and trace analytics inside the ELK stack rather than traditional BI.
[[Looker]]: Google Cloud’s enterprise BI where LookML defines a governed, code-based semantic layer that’s now exposed to Tableau, Power BI, and AI agents.
[[Rill Developer]]: Fast, BI-as-code dashboards defined in YAML + SQL with an embedded DuckDB/ClickHouse OLAP engine for sub-second queries.
[[Lightdash]]: Open-source BI that lives directly on top of your dbt project, turning dbt YAML models and metrics into a self-serve analytics layer.
[[whaly]]: French self-service BI platform positioned as a cheaper Looker alternative, focused on embedding analytics into CRMs, Slack, and field-sales apps.
[[Holistics]]: Analytics-as-code BI with its own AMQL semantic language, Git version control, and canvas-style narrative dashboards — a modern Looker alternative.
[[Lightdash]]: Open-source BI that lives directly on top of your dbt project, turning dbt YAML models and metrics into a self-serve analytics layer.
[[Redash (dashboarding on top of Delta Lake)]]: Lightweight open-source SQL-first tool — a browser-based query editor over 35+ data sources that turns queries into shareable charts and dashboards.
[[Mode]]: Collaborative analyst-first platform combining SQL, Python, and R notebooks with reports, now part of ThoughtSpot.
[[AtScale]]: Universal semantic layer that speaks SQL, MDX, and DAX natively, bringing OLAP-cube-style modeling (à la SSAS) to cloud warehouses without moving data.
[[Power BI]]: Microsoft’s BI platform with deepest Excel/Office 365 integration, DAX formula language, and Copilot AI baked into the broader Microsoft Fabric stack.
[[Toucan Toco]]: French “data storytelling” platform that prioritizes narrative-driven, mobile-first dashboards for non-technical business audiences.
**[[Evidence]]: BI as Code with a Markdown notebook style, rendering SQL-backed, NYT-style data reports as a static site from plain Markdown files.
[[Steep]]: Stockholm-based metrics-first BI where everything starts from a centrally defined metric catalog, built natively on the semantic layer (dbt Cloud, Cube).
[[Qlik]]: Known for its patented Associative Engine that in-memory-indexes all relationships so users can freely explore in any direction — including “unrelated” data.
[[Sigma (BI)]]: Warehouse-native BI with a familiar spreadsheet UI over Snowflake/BigQuery/Databricks, plus Input Tables that let users write back to the warehouse.
[[Observable Framework]]: Open-source static-site generator for data apps, using polyglot data loaders (any language) to precompute snapshots for instant-loading dashboards.
[[Omni]]: Founded by ex-Looker folks, combines a governed semantic layer with spreadsheet-like and SQL workflows so analysts and business users share one tool.
[[Shaper]]: Visualize and share your data. All in SQL. Powered by DuckDB.
[[Graphene]]: BI built for agents.
[[Mosaic]]: An extensible framework for linking databases and interactive views.

Libraries

[[MotherDuck Dives]]: An interactive visualizations you create with natural language, directly on top of your data in [[MotherDuck]].

Overview

![[Pasted image 20220922183401.png]]
Source

Declarative Data Pipelines

Wed, 03 Jun 2026 13:24:26 +0000

A declarative data pipeline is characterized by not specifying the execution order. Instead, it allows each step or task to autonomously determine the optimal time and method for execution.

QuackStore: OLAP Cache Layer for DuckDB

Wed, 03 Jun 2026 09:08:18 +0000

[[OLAP Cache Layer]] locally with QuackStore.

Created by [[Coginiti]].

Speed up your data queries by caching remote files locally. The QuackStore extension uses block-based caching to automatically store frequently accessed file portions in a local cache, dramatically reducing load times for repeated queries on the same data.

Microsoft Fabric

Tue, 02 Jun 2026 20:49:28 +0000

New announcement [[2023-05-23]] about the single place of MS; it’s like a [[Databricks]] replica, except for the tools in [[Azure]] without needing to go into Azure (see more below). See the announcement Introducing Microsoft Fabric and Copilot in Microsoft Power BI.

Bluesky

Tue, 02 Jun 2026 15:27:16 +0000

[[Twitter]] alternative. Created by the original founder of Twitter [[Jack Dorsey]].

The Bluesky (open source) implementation of [[Personal Data Server (PDS)|PDS]] uses [[SQLite DB|SQLite]] per user.

Open Social Media (and back to Open Web)

Tue, 02 Jun 2026 15:13:38 +0000

Open Social is a concept or idea to bring back the old [[internet]] with [[Personal Blogs]], and not give everything to “hosters” (social media platforms), as [[Dan Abramov]] explains. I wrote about it even before, I called it Inside the New Open Social Network.

Obsidian Webclipper

Tue, 02 Jun 2026 11:31:33 +0000

Obsidian Webcliper is an alternative to [[ReadWise]] web clipper.

Super powerful to clip websites to my [[Obsidian]] vault. Even with added AI-interpreted (e.g. description, scanning a LinkedIn Profile, etc.) for a summary. Find all the features in the documentation.

Taking Notes for Physical Books

Tue, 02 Jun 2026 11:25:15 +0000

Usually I’m listening to book on [[Audible]], but the workflow is the same.

Workflow of Taking Notes

I have two workflows related to reading [[Books]]:

DuckDB with Obsidian

Tue, 02 Jun 2026 10:58:07 +0000

Obsidian-duckdb-motherduck is a plugin for [[Obsidian]] that works similarly to [[Obsidian Dataview]], but with [[SQL]].

It can run arbitrary data with SQL, as DuckDB supports almost any format, you could query local CSVs, XML, or anything else. But not only local files work, but also remote files via [[DuckDB#Extensions|DuckDB’s Extensions]] such as HTTP(S), Huggin Face with hf: or anything else.

dbt (and dbt-core)

Tue, 02 Jun 2026 07:46:30 +0000

dbt (data build tool) allows you to establish macros and integrate other functions outside of [[SQL]]’s capabilities for advanced use cases. Macros in Jinja are pieces of code that can be used multiple times.

Slowing Down

Mon, 01 Jun 2026 10:58:05 +0000

Slowing down, like [[Thinking, Fast and Slow (Daniel Kahneman)]] and [[Slow Living]] is the best way to live life. My credo was always slowing down for more [[Long term game|long term]] impact, but that’s more true than ever?

Voice to Text (with AI)

Mon, 01 Jun 2026 09:25:03 +0000

In addition to [[Using Obsidian with AI|using AI in Obsidian]], I’m also allergic to voice-to-text.

E.g., if I do it in a car or so, I mostly need to spend time fixing or updating words that it wrongly interprets, or I also ramble much more when I talk than when I actually type. Again, [[writing is thinking]], so if you skip that, you also skip the thinking.

Writing in a Foreign Language: How English Gave Me My Voice

Fri, 29 May 2026 13:19:16 +0000

This is how I found [[My Writing Voice]] and style. I initially read a lot of English books because I didn’t know the language well when I was in [[Copenhagen|CPH]] and was learning it.

Creativity is Forward Looking

Thu, 28 May 2026 13:34:59 +0000

Remember what AI is, despite the fact that there are people in Silicon Valley who don’t want you to believe this is big data sets, lots of compute, and a large language model mushed together. That’s what they are.

'Having Taste' as an Artist—Age of AI

Thu, 28 May 2026 13:31:04 +0000

Having taste is key to have in the age of [[Generative AI]]. See the example of [[Rick Rubin]] in [[The Future with Generative AI]].

The Future with Generative AI

Thu, 28 May 2026 13:21:25 +0000

Unironically, Rick Rubin is the future of work. [[Tom White]] at Rick Rubin Is the Future of Work

[[Rick Rubin]] says as one of the most prolific music producer alive what: “I know what I like and what I don’t like, and I’m decisive about what I like and what I don’t like”. The future of [[Generative AI]] might just to [[Having Taste (with AI)]], and hearing to your [[Gut Feeling]], and make decision from the heart, like [[Writing from The heart]].

Analytics API

Thu, 28 May 2026 08:25:49 +0000

The struggle of having a single performant, secure, and reliable data endpoint is real. Especially to select the exact metric and dimension with the correct number everybody agrees on, right?

Cube

Wed, 27 May 2026 20:52:17 +0000

Formerly known as Cube.js, now simply Cube on Cube.dev. Cube is a [[Semantic Layer]] that is built as an [[OLAP]] cube capabilities but includes [[Analytics API]] capabilities too with fetching data with SQL, REST, and GraphQL out of the box.

Bed, Bath or/and Bus

Wed, 27 May 2026 20:42:09 +0000

Bed, Bath or Bus is shared by [[David Perell]] at NYT Bestselling Author Teaches Writing (Suleika Jaouad Interview), and a friend told him to relax his brain to get off a [[Writers Block]].

Writer's Block

Wed, 27 May 2026 20:30:07 +0000

As with [[Creativity vs Productivity]], I try to embrace the block and do some creative work instead. Writing is supposed to be hard. If you go on and get over the first 10 minutes ([[Ultradian Rhythm]]), you will get into the flow and [[Deep Work]] again.

Open Table Format Catalogs

Wed, 27 May 2026 13:14:44 +0000

Open Catalogs are similar to the [[Hive Metastore]] before, an index for what tables you have in your data lake.

Journaling

Wed, 27 May 2026 08:36:00 +0000

Another one that goes hand in hand with [[Stoic]] is Journaling. It grounds me similarly to meditating. I always heard on the [[Tim Ferriss]] Podcast that “successful” people meditate and journal a lot. I never understood the term journaling until I started it myself.

Church Business vs. Banking Business

Tue, 26 May 2026 08:22:20 +0000

Rick Rubin is in the Church Businesss, as it’s about emotions and selling a feeling, versus the mentioned man, is selling real things, probabilty etc.

Why Have Your Website

Tue, 26 May 2026 06:24:35 +0000

I’m saying this repeatedly: have your own website and domain. Platforms come and go. I have experienced it myself with Medium, WordPress, and inspired by Why does every personal website look like this now?, I wrote:

Skills: Evergreen Notes for Agents

Mon, 25 May 2026 19:54:37 +0000

Markdown files that explain an [[AI Agents]], how to use a skill. For example, you can explain how to run [[GoHugo]], what the features are, and explain everything well in a Markdown file. If you bundle these, you have a repo containing multiple “skills”.

Digital Minimalism

Mon, 25 May 2026 10:14:50 +0000

The elephant in the room is today’s distractions, such as social media apps that the most advanced technology companies make and Neuroscientists to make you addicted. In [[Digital Minimalism (Cal Newport)]], [[Cal Newport]] suggests that deleting all social media apps is already a good start as all companies will track you through the app. Instead, use the browser, they will have less or no GPS information when you open the app, cannot send you to push notifications, etc. You are already addicted, as hard as you want to tell yourself otherwise.

Extending SQL for Analytics

Mon, 25 May 2026 09:14:04 +0000

[[SQL]] is the origin of data. And it was extended over the years. We have [[geospatial]] capabilities and [[window functions]] and whatnot throughout the year, so why not have analytics semantics added, too?

Metrics

Mon, 25 May 2026 05:29:00 +0000

Metrics, [[KPI|KPIs]] (Key Performance Indicators), and Calculated Measures are terms forming the foundation of how business performance is measured and defined.

Data Engineering Whitepapers

Mon, 25 May 2026 05:27:51 +0000

A curated list of influential whitepapers in the field of data engineering.

Data Lakehouse

[[Data Lakehouse]] combines the best of data warehouses and data lakes into a single architecture.

Coginiti Semantic SQL

Fri, 22 May 2026 14:38:55 +0000

A SQL dialect supported by the [[Coginiti]] semantic layer. The [[Semantic Layer]] uses [[DataFusion|Apache DataFusion]] as its query engine and translates queries to platform-specific SQL for execution on your data warehouse.

Declarative Data Stack

Fri, 22 May 2026 14:24:00 +0000

Declarative Data Stack is a term introduced in the article The Rise of the Declarative Data Stack by [[Michael Driscoll|Mike Driscoll]] and myself.

The Vim Language (and Motions)

Thu, 21 May 2026 12:31:07 +0000

Why are there many users (and many new ones) still using [[Vim]] or [[Neovim]], you might ask?

It’s mainly for vim language/motions. Many believe, me included that you need to know 1000 shortcuts. But when I realized that Vim has a language that works like grammar, things changed. I noticed that I only have to remember a couple of modes (Copying (or yanking) in Vim with y, deleting with d, pasting with p, changing with c) and shortcuts, which I can then combine. This made me want to try it, and I didn’t regret it.

How to Review a PR without Leaving the Terminal (Neovim)

Thu, 21 May 2026 08:04:36 +0000

I made a video a while back with [[Neovim]] doing [[git difftool|git diff]]:

Foam (VSCode Extension)

Thu, 21 May 2026 07:15:06 +0000

Foam is another way of deploying backlinks from [[Obsidian]] open-sourced on my blog. Alternatives are found also in [[Open-Source Obsidian Publish Alternatives]].

Mindstone

Thu, 21 May 2026 07:02:59 +0000

This is a free alternative to [[Obsidian]] publish. Other once can be found in [[Open-Source Obsidian Publish Alternatives]].

MindStone is a free open-source alternative solution to Obsidian Publish.

Obsidian Digital Garden

Thu, 21 May 2026 07:01:34 +0000

This is a free alternative to [[Obsidian]] publish. Other once can be found in [[Open-Source Obsidian Publish Alternatives]].

Publish your notes in your own personal [[Digital Garden]]. An example can be found here.

One Big Table (OBT)

Tue, 19 May 2026 13:53:28 +0000

And extension of [[Ralph Kimball|Kimball]], suitable for long-term analysis. Essentially a [[Denormalization|denormalized]] table, akin to what is found in a [[Data Mart]]. And also synonym with a Wide Table.

Integrate OpenAI into Obsidian

Tue, 19 May 2026 11:52:22 +0000

[[OpenAI (GPT-3)]] is a powerful AI that can write whole articles, see Data Nets (fully written by AI). It’s the same AI backbone [[GPT-3]] that is used for [[GitHub Copilot]].

Social Media Addiction

Tue, 19 May 2026 06:09:25 +0000

This is my notes about [[Social Media]], and especially about the bad side of it, the [[addiction]].

The Bad: Big Tech Makes Money

How a handful of tech companies control billions of minds every day and how Big Tech is dangerous.

GitHub Alternatives

Wed, 13 May 2026 13:41:44 +0000

[[GitHub]] is the main place to host [[git]] versioned code as of now. But there are alternatives not owned by [[Microsoft]], and had many outtages and failures lately.

Reading Books for a Happy Life

Tue, 12 May 2026 07:15:15 +0000

Instead of reading depressing news, what will make you happy is reading books. If you are not a bookworm or not reading generally, you won’t agree with this. But as someone who didn’t read at all or was slow for a long time, I can tell you it’s true. By reading, you enlarge your horizon, you learn something new, or if you read a fantasy book, you dive into another world in an undistracted, calming way.

Audiobooks

Tue, 12 May 2026 07:07:10 +0000

[[Books]] recorded as audible, to be consumed via voice. Usually on [[Audible]] or similar services.

I almost exclusively read audiobooks. Due to my work, spending all day reading on a screen, I discovered that my eyes are usually very tired, but my brain is still able to listen. For example, relaxing in bed, reading a book, and my eyes fall asleep. But instead, I can close my eyes and listen to a good audiobook for 30 to 45 minutes before I sleep with no problem.

Idempotency

Mon, 11 May 2026 11:10:02 +0000

Idempotency is the property of a certain operation that can be applied multiple times without changing the resulting outcome by being given the same inputs. It is used in [[Functional Programming]] and was the base foundation for [[Functional Data Engineering]].

Slow Technology: Better Productivity through Slowing Down?

Mon, 11 May 2026 07:18:42 +0000

Slow technology to me is technology that is slow in the moment, but increases [[productivity]] in the [[Long term game|long term]], as in the slow process and the [[slowing down]], makes you [[Deep Work|focus]] and creates more insightful outcomes. And let’s you create more meaningful and more [[Deep Work]].

Dremio

Mon, 11 May 2026 07:00:47 +0000

Dremio revolutionizes self-service analytics for data teams. It empowers data analysts to explore and visualize data with rapid query response times. Concurrently, data engineers benefit from the ability to ingest and transform data directly in the data lake, fully supporting DML operations.

Second Brain Assistant with Obsidian (NoteGPT)

Fri, 08 May 2026 14:14:53 +0000

Harnessing the profound capabilities of my Second Brain, I’ve always envisioned the integration of an [[OpenAI]] model directly into my local files. The universe of open-source solutions beckons.

Functional Data Engineering

Fri, 08 May 2026 04:53:32 +0000

Functional data engineering is crucial for designing data engineering architectures, bringing [[clarity]] with “pure” functions and removing side effects.

Its [[Functional Programming]] applied to the field of data engineering initiated by [[Maxime Beauchemin]]. Functional Data Engineering is a paradigm for batch data processing.

Granularity

Tue, 05 May 2026 15:02:17 +0000

In [[Dimensional Modeling]], defining the grain is a pivotal step. This process involves determining what each row in a fact table signifies, laying down a foundational framework for the entire design.

File Over App

Tue, 05 May 2026 09:43:54 +0000

File over app is a philosophy: if you want to create digital artifacts that last, they must be files you can control, in formats that are easy to retrieve and read. Use tools that give you this freedom.

AI Whitepapers

Tue, 05 May 2026 08:36:29 +0000

Similar to [[Data Engineering Whitepapers]] is here the papers related to [[Artificial intelligence (AI)|AI]] and [[AI Agents]].

Large Language Models

[[Large Language Models (LLMs)|LLMs]] are the general intelligence made out of large [[neural network]].

Modern Data Stack

Tue, 05 May 2026 07:16:04 +0000

The Modern Data Stack (MDS) comprises a suite of open-source tools designed for end-to-end analytics. This includes data ingestion, transformation, machine learning, and integration into a columnar data warehouse or lake solution, all complemented by an analytics BI dashboard backend. The stack’s versatility allows extensions for data quality, data cataloging, and more.

Import Bible to Obsidian (Markdown)

Mon, 04 May 2026 07:57:42 +0000

Below is how I imported the Bible into [[Obsidian]] as [[Markdown]] files with the help of this repository: BibleGateway-to-Markdown, a way of parsing NET Bible passage output from Bible Gateway.

Extremely Personal Software

Mon, 04 May 2026 07:06:32 +0000

The next phase of [[self-hosting]], building your own software for thinkgs you need.

Personal Software now possible via vibe coding

For example, this gentlemen have all tools made by himself, even [[vim]].
![[img_Extremely Personal Software_1777878279371.webp]]

Parametric Data Pipeline

Fri, 01 May 2026 13:28:08 +0000

Once the data is normalized into the unified model, the next component is the parametric pipeline. This [[data pipelines|pipeline]] is designed to transform the normalized data into datasets that enable meaningful insights, metrics, and KPIs. The term “parametric” refers to the ability of the pipeline to adapt to various configurations and customizations through parameters, while adapting the underlying logic.

SQL

Fri, 01 May 2026 12:26:02 +0000

SQL, a domain-specific language, is pivotal in programming, especially for managing data in relational database management systems or stream processing in relational data stream systems.

SQL IDEs

Fri, 01 May 2026 09:43:34 +0000

SQL [[IDE|IDEs]] are the butter and bread of a [[Business Intelligence Engineer]] and [[Data Engineer]].

Terminal based

[[Query databases in vim (vim-dadbod)|vim-dadbod]] ( GitHub) - see [[Query databases in vim (vim-dadbod)]]
- Supports: Big Query, [[ClickHouse]], Impala, jq, MongoDB, MySQL, Oracle, osquery, PostgreSQL, Presto, Redis, SQL Server, [[SQLite]], DuckDB
- See a short video of how I use it with DuckDB on asciinema):
[[Harlequin]]: Started with [[DuckDB]] and SQLite
Rainfrog: A database tool for the terminal.
sqlit: - A user friendly TUI for SQL databases.

Tools (Free)

[[DBeaver]]: Highly versatile and supports numerous databases. Its open-source nature and extensive functionality make it a top choice for many professionals.
[[HeidiSQL]]: A lightweight tool that’s great for quick and efficient work with MySQL, MariaDB, and Microsoft SQL Server databases.
[[MySQL Workbench]]: Tailored for MySQL databases, it offers a complete set of tools for database design, development, and administration.
[[SQL Server Management Studio (SSMS)]]: A go-to for those working primarily with [[Microsoft SQL Server]]. It offers comprehensive tools for configuration, management, and SQL development.
[[Oracle SQL Developer]]: Designed specifically for Oracle Database, it provides a comprehensive development and management toolset.
[[PgAdmin]]: Widely used by PostgreSQL database developers, it’s an open-source tool that offers a rich set of features for database management.
Aqua Data Studio: Offers database management, development, and visualization tools for a variety of databases.
[[Toad]]: Known for its database management tools, it supports Oracle, SQL Server, MySQL, and other databases. Toad is particularly popular for its automation features and extensive functionality.
Valentina Studio: A multi-platform tool that works with MySQL, PostgreSQL, MariaDB, and SQLite. It offers excellent schema editing and diagramming capabilities.
DbForge Studio: A comprehensive IDE for SQL Server, MySQL, Oracle, and PostgreSQL. Offers a wide range of tools for database development, management, and administration, including data comparison and synchronization, database diagramming, and SQL code completion.
Aqua Data Studio: An advanced IDE that provides database developers with tools for editing, querying, and managing various databases. It’s known for its sophisticated query tool and versatile database support, including visual analytics.
SQLyog: A powerful MySQL manager and admin tool, SQLyog is known for its intuitive GUI and a rich set of features like schema synchronization, query builder, and data transfer tools.
DataSpell: Developed by [[JetBrains]], it’s a new IDE specifically designed for data science, offering a productive environment for Python and R with integrated [[Jupyter Notebook]] support.
VSCode Extensions [[VSCode|Visual Studio Code]], a lightweight but powerful source code editor, offers SQL development extensions. These extensions add database connectivity, query execution, and result set viewing capabilities to VSCode.

Payed

[[TablePlus]]: A modern, native tool with a clean user interface that allows developers to simultaneously manage multiple databases such as MySQL, PostgreSQL, SQLite, and more. It’s known for its simplicity and robustness.
[[DataGrip]]: Developed by JetBrains, it’s known for its intelligent query console, schema navigation, and version control integration. It’s a powerful tool, especially if you’re working with multiple databases and SQL dialects.
- since [[2025-10-01]] its free for none commercial use. Announcement
Navicat: A series of database management products that support MySQL, MariaDB, SQL Server, Oracle, PostgreSQL, and SQLite. It offers an intuitive interface with powerful tools for data editing, SQL query and report building, and backup management.
Aginity Pro: A specialized tool primarily focused on enhancing productivity for SQL data analysts and developers. It provides features for code reuse, query optimization, and team collaboration.
Azure Data Studio: [[VSCode]] like SQL editor.
Coginiti Pro by [[Coginiti]]: SQL Analytics Tool and Reporting Platform

Origin: [[Harlequin]]
References: SQL IDE of choice? : r/dataengineering, [[SQL]]

Dimensions

Wed, 29 Apr 2026 20:43:57 +0000

A dimension is a structure that categorizes and describes data in a business process to enable easy access and analysis in [[Dimensional Modeling]].

Physical Data Model: Bottom-Up Approach

Wed, 29 Apr 2026 19:13:35 +0000

The physical model involves implementing or generating the model in the target database system, taking into account the unique syntax variations of each database.

Conceptual Data Model (CDM): A Top-Down Approach

Wed, 29 Apr 2026 19:12:14 +0000

In the top-down approach, we commence with a high-level perspective of an organization’s data requirements. This process involves collaborating with business owners, domain experts, and various stakeholders to grasp the business necessities and formulate a conceptual data model.

Bus Matrix

Wed, 29 Apr 2026 18:11:32 +0000

A Bus Matrix was traditionally used in [[OLAP]] cubes such as Microsoft [[SSAS]] and co. They let you visually see what [[Measures]] can be queried with which [[Dimensions]].

Base Lists

Wed, 29 Apr 2026 06:36:27 +0000

My favorite Coffee Beans List
My favorite Book List
[[_Food]]

Origin: My [[Obsidian Base]] lists.

Apache Arrow

Tue, 28 Apr 2026 10:32:58 +0000

Apache Arrow is [[in-memory formats|in-memory analytics format]]. It serves as a robust development platform for in-memory analytics. It encompasses a suite of technologies designed to expedite the processing and transfer of big data.

Walker Launcher

Mon, 27 Apr 2026 12:50:15 +0000

Walker, is a [[Raycast]] alternative for Linux to launch apps.

It has many more features included - I’d say if you search for a [[Raycast]] solution on Linux, search no more. It’s amazing how Walker provides file search, clipboard, calculator, emojis, and many more out of the box. It does more than I needed Raycast to do.

Advanced search and Replace (like VSCode) in Neovim (with `:cdo`)

Mon, 27 Apr 2026 10:33:12 +0000

Search replace is a key need in [[IDE]]s, but Neovim does it a bit different than usual editors. That’s why I created below notes for myself to solve the same with :cdo and QuickFix lists.

Claude Code

Mon, 27 Apr 2026 08:38:51 +0000

This is an [[AI Agents|Agentic Coding Tool]] that reasons and writes code.

Improved AI-Workflow

To be, most important is still the improved [[AI workflow]], not the autocomplete of [[AI-based IDE]], and that’s where Claude Code is doing everything right. With it’s powerful [[AI Agents|Agentic Coding Tool]] style, it’s super powerful to create todo list for itself, interpreting existing code and working its way throught it, autonomously.

Gut Feeling and Intuition

Mon, 27 Apr 2026 08:15:31 +0000

The gut feeling or Intuition is an instinctive feeling, as opposed to an opinion based on facts.

Intuition can only be learned through [[experience]]. You can’t take shortcuts. It makes all the difference at work between an exceptional developer and a junior who didn’t have that past experience, hence training his gut and intuition.

Ideas Worth Sharing

Mon, 27 Apr 2026 08:15:31 +0000

Great insights are coming from having lived. You can’t share great ideas out of thin air. Either you had a hard time, you have lived in different places ([[Mini Retirements]]), you encountered many problems and overcame obstacles. These all make great insights, articles to share.

What Makes a Happy Life

Mon, 27 Apr 2026 07:47:12 +0000

Besides [[Stoic|Stoicism]] and [[Journaling]], it’s relationships. According to the most comprehensive study in happiness, the happiest people are the ones having genuine relationships.

Deep Work

Sat, 25 Apr 2026 07:58:00 +0000

Or also called flow. Introduced in the book [[Deep Work (Cal Newport)]].

Deep Work is an indispensable guide to anyone seeking focus. One of the most valuable skills in our economy is becoming increasingly rare. If you master this skill, you’ll achieve extraordinary results.

Programming Languages

Fri, 24 Apr 2026 06:32:51 +0000

We have [[imperative]] and [[declarative]] languages. (see [[Declarative vs Imperative]])
We have [[Functional Programming Language]]. (see [[Functional Data Engineering]])

Languages

[[C (Programming Language)]]
[[C++]]
[[Rust]]
[[PHP]]
[[Python]]
[[JavaScript]] / [[TypeScript]]
[[Java]]
[[GoLang]]
[[Ruby On Rails]] (joy)
[[Clojure]]
…..
Newer ones:
- [[Odin]] (joy)
- [[Zig|Ziglang]]
- [[Jai (Programming Langauge]] (joy)

Functional Ones

[[Functional Programming Language]]

Fonts

Fri, 24 Apr 2026 06:26:10 +0000

I have a love for nice fonts.

Reading and Writing Fonts

[[Atkinson Hyperlegible]] (mentioned Your favourite fonts choice and why? thread - Share & showcase - Obsidian Forum )
iA Writer Quattro
Inter - Default of [[Obsidian]] and used also at [[Airbyte]] website
[[Fonts - DYSLEXIA]] which makes reading easier for people with problems (from [[Tobias Ziegler (Schäuble)]])
Piazzolla, serif; from How to treat monospace alike fonts in a tech blog? - Pimp my Type
- Piazzolla Type System
Gentium Book Basic, serif: Discovered on Why Write?, beautiful .
Also beautiful: SN Pro ( origin)
UxumGrotesque, sans-serif ( origin)
- ![[Fonts-20240905094328551.webp]]

Atkinson Hyperlegible vs Piazzolla:
![[Pasted image 20221022164222.png]]

Open Standards over Silos

Thu, 23 Apr 2026 18:58:47 +0000

Great article on [[RW Open standards over silos Voltron]] by [[Voltron Data]]. It reminds me of [[File Over App]] by [[Kepano]].

When AI Becomes Addictive

Mon, 20 Apr 2026 19:23:42 +0000

I think [[Social Media Addiction]] is a big thing, and I’m noticing more that AI and [[Generative AI]] are facing the same problems.

Adrianos Liberica

Fri, 17 Apr 2026 11:25:40 +0000

Link: Liberica Kaffee Bohnen

![[Adrianos dd-20241009112558485.webp]]

Origin:

KAFOJ Espresso

Fri, 17 Apr 2026 11:12:39 +0000

Biel coffee More.

Origin: visited [[2025-02-08]], tried this coffee today at the shop
References: [[KAFOJ]], similar to [[Adrianos Espresso]]

Adrianos Espresso

Fri, 17 Apr 2026 10:48:04 +0000

![[esprsso.png]]

My everyday choice.

Origin: [[Coffee Beans (dataview)]]

Automated OG Image Generation: Rust + SVG → WebP

Fri, 17 Apr 2026 10:32:39 +0000

For [[My Second Brain]] I create programmatical way of generating OG (Open Graph) / hero image. These get’s generated by adding the title of the note to the [[SVG]], but otherwise use a og_template.svg as a template, then converts them to WebP format for optimal web performance.

Problems of Business Intelligence

Thu, 16 Apr 2026 19:38:40 +0000

On the other hand, BI has some substantial problems with speed and transparency. I tried to summarize the issues I learned or heard people telling over my career as a BI engineer and specialist working with Oracle and SQL Server:

tmux

Thu, 16 Apr 2026 08:39:36 +0000

tmux is an open-source [[Terminal Multiplexer]] for Unix-like operating systems. It allows multiple terminal sessions to be accessed simultaneously in a single window. It is useful for running more than one command-line program at the same time.

Apache Airflow

Thu, 16 Apr 2026 08:13:54 +0000

As outlined in Orchestrators, the scheduling and monitoring of workflows stand as pivotal choices. While there are numerous [[Data Orchestrators|Orchestrators]], Airflow emerges as the most prevalent and acclaimed.

Distract-Free Typewriter

Mon, 13 Apr 2026 11:19:36 +0000

Distraction-free Writing Device with Full keyboard and ePaper Display. A modern Typewriter.

![[img-Distract-Free Typewriter-1740409001580.webp]]
Source: Building the Micro Journal

Another great take on a modern typewriter. Instead of Multi-tasking, focus on the «Unitaskers».

Deep Life

Mon, 13 Apr 2026 11:03:59 +0000

Deep life, and productivity on their own do not work. [[Cal Newport]].

Besides a sound [[Personal Knowledge Management (PKM)]], a place to store all your knowledge and handle your to-dos, much more is needed for a fulfilled life. Most of them happen inside my PKM system. That’s why it’s strongly connected with a deep life, which helped me keep grounded and to have a better meaningful life, similar to what the PKM system does to me.

The State of DevOps in Data Engineering

Thu, 09 Apr 2026 11:44:54 +0000

Is [[DevOps]] the new data engineering of data science? What do I mean? As in the old days, when you were doing data science but spent 80% of your time on data engineering.

Kubernetes

Thu, 09 Apr 2026 11:44:41 +0000

It’s a platform that allows you to run and orchestrate container workloads. Kubernetes has become the de-facto standard for your cloud-native apps to (auto-) [[Horizontal Scaling|Scale-out]] and deploys your open-source zoo fast, cloud-provider-independent. No lock-in here. You could use [[OpenShift]] or OKD. With the latest version, they added the OperatorHub which you can install as of today 182 items with just a few clicks. Also, check out [[Managed Data Stacks]] which were created to mitigate exactly that.

Admonition / Callouts (Markdown)

Thu, 09 Apr 2026 08:41:36 +0000

The admonition, also known as call-outs, are shortcuts baked into most Markdown framework. It’s a nice way to include side content without significantly interrupting the reading flow.

Archetypes of Notetakers

Thu, 09 Apr 2026 07:27:29 +0000

[[Anne-Laure Le Cunff]] considers three types of note takers: architects, gardeners and librarians. This is what she means by this.

Shallow Work: Be Aware of Striving Over-Productivity

Thu, 09 Apr 2026 07:12:56 +0000

Be aware and careful about «Shallow Work» (also called pseudo work or meta-work).

With all the AI hype and over-[[productivity]] we (suppose) to get with [[OpenClaw]] and others, we need to be careful not to do pseudo-work all day long and not finish our actual tasks.

Mentorship in Times of AI

Thu, 09 Apr 2026 05:50:54 +0000

I’ve read the article Would I have been mentored in a world full of AIs? by [[Marcus Coetzee]] and it got me thinking.

Four Modes of Writing

Wed, 08 Apr 2026 07:56:22 +0000

My [[writing]] process has [[Four Modes of Writing|four modes]].

NORMAL mode: jumping around, reading, learning
INSERT mode: writing, thinking, making connections
VISUAL mode: copying, highlighting, formatting, designing
COMMAND mode: automate, fix, macros.

In case you can’t tell, I use [[Vim Language (and Motions)|Vim Motions]] in [[Obsidian]] 😉.

HEY-Screener in (Neo)Mutt

Thu, 02 Apr 2026 13:08:39 +0000

To create my favorite feature from HEY.com, the [[Screener]], I had to get the existing work and screen in and out of the list for two years or so. Below is how I made it.

Rubiks Cube: How to solve it

Wed, 01 Apr 2026 09:28:02 +0000

What challenged me to solve one myself: There are so many possible configurations (over 43 quintillions) that it would be impossible to solve it by randomly turning the faces until it’s all done.

Kinesis Advantages 2: Lubing and Dampening

Fri, 27 Mar 2026 12:25:38 +0000

After a [[Ducky One 2 SF]] and then my first build, a [[My KBDFans DZ65RGB|KBDFans DZ65RGB]], I have to say my first impression wasn’t good. The keys are wabbling, the switches are scratchy, and the sound is very hollow. And of top, the software to program any keys is crap. You can’t program a combination of two keys mapped to one, and there is essentially no second layer that you can shift with a key while holding, only toggle until you toggle it back.

Dotfiles

Fri, 27 Mar 2026 07:42:05 +0000

Dotfiles are important files that will play an integral role in your career as a software developer. They are configuration files for various programs that help those programs manage their functionality.

SQL Server Analysis Services (SSAS)

Thu, 26 Mar 2026 16:01:19 +0000

[[OLAP]] cubes in Service Manager leverage the existing data warehouse infrastructure to empower end users with self-service [[business intelligence]] capabilities.

Sharing as Second Brain Note vs. a Blog Post

Wed, 25 Mar 2026 15:30:47 +0000

I like to collect notes over time on my [[Second Brain]], and eventually, when I figure something substantial to share, something I learned, I write a blog post about it.

HEY.com: Get a list of ScreenedIn/Out emails

Wed, 25 Mar 2026 07:25:31 +0000

[!note]

If you are here, how the workflow works, there will be an upcoming note [[HEY-Screener in (Neo)Mutt]], but for now, you can check all scripts on my Mutt dotfiles.

AI-based IDE

Mon, 23 Mar 2026 16:38:04 +0000

Same as [[IDE]]s, but AI based.

Tools

[[Cursor AI IDE]]
[[Windsurf AI IDE]]
aider

Terminal based

See tools at [[Vibe Code Agents]].

Shallow Happiness

Mon, 23 Mar 2026 16:33:50 +0000

Opposite to [[Deep Happiness]]. Find more on my article about Finding Flow: Escaping Digital Distractions Through Deep Work and Slow Living ([[Stolen Focus- Why You Can’t Pay Attention (Johann Hari)|internal]]).

Metrics Layer

Mon, 23 Mar 2026 12:26:21 +0000

[!info] Similarities to a Semantic Layer

The metrics layer is a key component of a [[Semantic Layer]]. Often, a basic metrics layer is integrated within a BI tool, translating its [[Metrics]] specifically for that BI tool.

Compounding Note-Taking

Mon, 23 Mar 2026 11:12:00 +0000

Note-taking is like saving money with interest. You save not only the interests of each year but also interest from interests. The same happens if you start taking notes; ideas will compound. Your notes will not only be worth their years but multiplied. My Tweet

How to Write Well?

Mon, 23 Mar 2026 10:43:27 +0000

It’s hard, it’s painful as Stephen mentions in his book [[On Writing (Stephen King)|On Writing]] (especially [[On Writing (Stephen King)#GOOD WRITING|good writing]]) and [[The Elements of Style (William Strunk Jr)]].

Two Minutes Rules

Mon, 23 Mar 2026 10:43:27 +0000

Introduced in the [[Atomic Habits (James Clear)]] book.

It says that:

Make the habit for at longest 2 minutes! You might do some more.
1. if it really doesn’t stick, stop after two minutes of the gym, for example.
The hard part is starting a habit. e.g., changing clothes before going to run. Running is the easy part; if you have changed, you will go no matter what

More on How to Stop Procrastinating by Using the “2-Minute Rule”.

Compound: Everything Does

Mon, 23 Mar 2026 10:28:27 +0000

Compounding is something so substantial. You can do it with notes ([[Compounding Note-Taking]]), money, and in life ([[Principles]]). It is slow, but it’s the right way to grow.

ETL vs ELT

Mon, 23 Mar 2026 08:42:15 +0000

[[ETL]] (Extract, Transform, and Load) and [[ELT]] (Extract, Load, and Transform) are key processes in data handling, with ETL occurring within a Data Warehouse and ELT within a Data Lake (refer to [[Data Warehouse vs Data Lake]] for more).

Relational Databases

Thu, 19 Mar 2026 13:03:53 +0000

In contrast to the [[Relational Model]], a relational database is a practical implementation of the relational model. Also called RDBMS - Relational Database Management System.

GoHugo

Wed, 18 Mar 2026 17:56:19 +0000

The world’s fastest framework for building websites. Hugo is one of the most popular [[Open Source]] [[Static Site Generators (SSG)]]. With its incredible speed and flexibility, Hugo makes building websites fun again.

Apache Parquet

Wed, 18 Mar 2026 07:46:23 +0000

Apache Parquet is a free and open-source column-oriented [[Data Lake File Formats|Data Lake File Format]] within the Apache [[Hadoop]] ecosystem. It operates similarly to RCFile and [[ORC]], other columnar storage file formats in Hadoop, ensuring compatibility with most data processing frameworks associated with Hadoop.

Data Engineering Blogs & Newsletters

Tue, 17 Mar 2026 22:02:23 +0000

This is a curated list of data engineering blogs.

Personal Blogs

My humble blog - ssp.sh: Technical Blog focusing on genuine news about the data ecosystem. Includes «.[[Data Engineering|Data Engineering Vault]]», [[Second Brain]], and a book about Patterns of Data Engineering.
Start Data Engineering by [[Joseph Machado]]
Confessions of a Data Guy by [[Daniel Beach]]
Eckerson Group by [[Wayne Eckerson]]
Software Engineering, Linux, Data, GIS by [[Christian Hollinger]]
SeattleDataGuy’s Substack by [[Ben Rogojan|SeattleDataGuy]]

Newsletters and Substacks

Blef by [[Christophe Blefari]]: Best data newsletter there is, IMO.
From An Engineer Sight by [[Benoit Pimpaud]]
group by 1 by [[Matt Arderne]]
SeattleDataGuy’s Newsletter by [[Ben Rogojan]]
Data People Etc. by [[Stephen Bailey]]
Joe Reis Substack by [[Joe Reis]]
Benn Substack by [[Benn Stancil]]
Petr Substack by [[Petr Janda]]
Pedram’s Data Based by [[Pedram Navid]]
Modern Data Democracy by [[JP Monteiro]]
GrouoBy Substack by [[Vu Trinh]]: Weekly curated data engineering resource and learning insights digesting big company blogs.
[[Newsletter|SELECT Insights]] by me

Find more blogs that I personally follow on [[RSS feeds for Data Engineering]], or people in the space at [[People of Data Engineering]].

Delta Load a Data Warehouse

Tue, 17 Mar 2026 21:06:55 +0000

Delta load is the process of loading data from your source OLTP database, the operational database where the application is running. This database is modeled in a relational fashion, normalized into 100s of database tables. With the delta load, we want to have an efficient way of transferring these data into the [[data warehouse]], modeling, and transforming the data so we can extract critical business metrics.

Load incremental and Idempotency

Tue, 17 Mar 2026 21:03:56 +0000

Compared to the nightly loads of traditional data warehouses, we need to load incrementally. This makes your data more modular and manageable, especially when you have a [[Star Schema]]. Fact tables can only be appended and dimensions only need to scan the newest transactions instead of the entire fact table.

Data Engineering Best Practices (Richard Schiller, David Larochelle)

Mon, 16 Mar 2026 20:16:43 +0000

Subtitle:
Link:
Tags / Categories: #📬 #📚 #reading
Date: [[2024-11-04]]
Author: [[Richard J. Schiller]], [[David Larochelle]]
Type: [[Books]]
Genre: #self-help #publish
Related: [[Data Engineering Best Practices]]
Started reading: not yet
Finished reading:
Origin:

[!summary]

Entity Relationship Diagram (ERD)

Mon, 16 Mar 2026 15:18:11 +0000

An Entity Relationship (ER) Diagram is a type of flowchart that illustrates how “[[Entity|Entities]]” (usually tables within a database) such as people, objects, or concepts relate to each other within a system.

Understanding the Difference: Writing within an App vs. a Note app

Fri, 13 Mar 2026 12:23:54 +0000

The difference between writing within the app you publish, let’s say, Twitter, LinkedIn, or the newsletter in [[Substack]], [[Medium]], [[Email]], etc., and writing it in [[Obsidian]] or a note app can make a big difference, but for different reasons.

MinIO

Thu, 12 Mar 2026 14:17:51 +0000

MinIO is an [[Storage Layer|object storage]] solution that provides an Amazon Web Services [[AWS S3|S3]]-compatible API and supports all core S3 features.

Master Data (Referenced or Stammdaten)

Wed, 11 Mar 2026 21:32:19 +0000

Master data is handled by [[Master Data Management]]. Typical master data are tables such as customer, product, address — things that should be managed centrally and updated once for all systems.

Transactional Data (Bewegungsdaten)

Wed, 11 Mar 2026 21:29:50 +0000

In [[Data Management]], dynamic data or transactional data is information that is periodically updated, meaning it changes asynchronously over time as new information becomes available.

WebAssembly (WASM)

Wed, 11 Mar 2026 16:41:51 +0000

WebAssembly (Wasm) is an open standard that enables the execution of binary code on the web. This format allows developers to leverage the performance of languages like C, C++, and [[Rust]] in web development. Currently, it is primarily used for demanding operations in the browser. For instance, [[Figma]] is powered by Wasm.

Why I Don't Research

Mon, 09 Mar 2026 18:45:12 +0000

and write from experience.

[!note] Published on my blog

This post has been published as blog post at Why I Don’t Research; and Write from Experience, but gets updated here.

Weekly Lists

Mon, 09 Mar 2026 08:43:35 +0000

Similar to what [[Kepano]] said, The only to-dos method that has worked for me is weekly lists:

Every week I make a weekly note, and write my to-dos for the week. I may add more items to it during the week.

How I write Daily Notes differently

Mon, 09 Mar 2026 08:42:47 +0000

Many are writing [[Daily Notes]]. I tried several times, but it never stuck. I think it’s because it’s too much work and too cluttered. For a look at my [[Daily Notes#My Daily Note Template]], there are some critical questions about gratitude. It lets me focus on yesterday’s todos and what I want to achieve tomorrow.

Todo's: Need Constraints

Mon, 09 Mar 2026 08:40:44 +0000

One thing I learned about todos, it’s never about the [[Todo Apps|todo app]], it’s about having constraints to not add anything immediately. Important things will resurface again. Urgent matters in the moment, but irrelevant tomorrow, will save us a lot of time.

Obsidian Plugins

Mon, 09 Mar 2026 08:16:28 +0000

Powerful [[Obsidian]] plugins.

Core

[[Obsidian Canvas]]
[[Obsidian Hotkeys]]

My Favorite Plugins

[!note] My used Plugin right now

All my settings of Obsidian are on GitHub, so you can always find which community plugins I have activated.

Open-Specific Files with Obsidian and Neovim

Mon, 09 Mar 2026 08:16:27 +0000

I wanted to open my todo or daily note with a single vim-shortcut or also hotkey from Obsidian.

There was not possible, so I create a very simple plugin. Check it out at: Obsidian-open-specific-notes (A small and simple plugin for Obsidian to quickly open a specific note (e.g. todo, daily journal).

SQL Server Reporting Services (SSRS)

Fri, 06 Mar 2026 21:12:35 +0000

SQL Server Reporting Services (SSRS) is a server-based report generating software system from Microsoft. It is part of a suite of [[Microsoft SQL Server]] services, including [[SSAS]] (SQL Server Analysis Services) and [[SSIS]] (SQL Server Integration Services).

Deep Happiness

Fri, 06 Mar 2026 19:00:26 +0000

The opposite of [[Shallow Happiness]]. Find more on my article about [[Stolen Focus- Why You Can’t Pay Attention (Johann Hari)]].

Orchestration

Fri, 06 Mar 2026 10:07:45 +0000

Delving deeper, after selecting technology from the myriad available, you’ll inevitably confront the need to manage intermediate levels. This is particularly true when handling unstructured data, which necessitates transformation into a structured format. Orchestrators and cloud-computing frameworks play a crucial role in this process, ensuring efficient data manipulation across different systems and formats. In the following chapters, I’ll elucidate their role in completing the full architectural picture.

Data Virtualization / Data Federation

Thu, 05 Mar 2026 20:45:27 +0000

Data Virtualization is particularly useful when you have multiple source systems from different technologies, all with relatively fast response times. If you don’t run many operational applications, you might consider Data Virtualization. This approach allows you to avoid moving and copying data around or pre-aggregating it. Instead, you create a [[Semantic Layer]] where you build your business models (like cubes), and queries to this data virtualization layer are then directed to the appropriate data source.

SecondBrain Disclaimer

Thu, 05 Mar 2026 09:25:12 +0000

A few words on how this public second brain is set up. Be aware that these are essentially my notes shared publicly. So expect incompleteness, grammatical errors, or sometimes broken links, etc.

Astro

Wed, 04 Mar 2026 18:21:31 +0000

A [[Static Site Generators (SSG)]], probably most known due to “ Dynamic Island” on Apples presentation.

History

Joined [[Cloudflare]] [[2026-01-16]], see Announcement

Personal Knowledge Management Workflow for a Deeper Life - As a Computer Scientist

Wed, 04 Mar 2026 11:13:48 +0000

![[PKM worflow mindmap.png]]

With burnout and mental stress at every level of our lives, I find my [[Personal Knowledge Management (PKM)]] system even more valuable. As a human, I forget lots of things. As a dad, I have more responsibilities with remembering all things related to my kid. As a developer and knowledge worker, I re-use code snippets or create new things. That’s why a PKM system such as a Second Brain to store all of it in a sustainable way is crucial to me.

Omarchy: An Opinionated Arch/Hyprland Setup

Tue, 03 Mar 2026 15:22:36 +0000

Omarchy turns a fresh [[Arch Linux]] installation into a fully-configured, beautiful, and modern web development system based on [[Hyprland]] by running a single command. Created by [[David Heinemeier Hansson (DHH)|DHH]] and hosted on GitHub.

Learning with AI

Tue, 03 Mar 2026 13:52:02 +0000

[[ChatGPT]] and other [[Large Language Models (LLMs)|LLM]] tend to disallow people to think and learn. My main concern is, if you use LLMs all the time, you run on auto pilot, and you tend to not learn and not think anymore (see more on [[Will AI replace Humans|Will AI Replace Human Thinking]]).

Open Subscription Platforms

Tue, 03 Mar 2026 10:05:59 +0000

Here, you see all platforms that exist as open: Open Subscription Platforms, and which ones are not.

[[Substack]]
[[Ghost]]
[[Medium]]
[[Buttondown]] for Emails only
[[Hashnode]]
[[ConvertKit]], [[Substack]], etc., see more tools on [[E-Mail Automation]]
[[Beehiiv]]
[[Memberstack]] (What I use for my book)
[[Gumroad]]
[[Listmonk]] (Newsletter I use for mine)
New once on top of [[AT Protocol]]. See below [[#Built on Top of AT Protocol]]

Others, self-made, are [[Static Site Generators (SSG)]], e.g. [[GoHugo]] where I build my second brain and website. Also, check out [[E-Mail Automation]].

Writing from the Heart

Tue, 03 Mar 2026 07:50:50 +0000

Whenever I have some time on vacation, I like to write something about my current interest, without much planning, writing the article within a couple of hours (vs. days and weeks I usually spend).

Books of Data Engineering

Thu, 26 Feb 2026 19:38:30 +0000

Below are books related to data engineering.

Starters and general knowledge:

Patterns of Data Engineering (PoDE) - My partially free online book to get you started :)
The Data Warehouse Toolkit, [[Ralph Kimball]]: February 1996, The guide for data warehouse modeling and still relevant to this day.
Designing Data-Intensive Applications by [[Martin Kleppmann]]
Fundamentals of Data Engineering, [[Joe Reis]], [[Matt Housley]]

Other great ones, more specific:

GoatCounter

Mon, 23 Feb 2026 19:34:44 +0000

Easy web analytics. No tracking of personal data.

GoatCounter is an open-source web analytics platform available as a free donation-supported hosted service or self-hosted app. It aims to offer easy-to-use and meaningful privacy-friendly web analytics as an alternative to [[Google Analytics]], [[Matomo]], or [[Umami (Web Analytics)]].

Great Open-Source Tools in Rust

Mon, 23 Feb 2026 13:44:30 +0000

As [[Rust]] gets more popular. Here are three open-source tools being exited to follow. ^14914f

Let’s see other open-source tools and frameworks built in Rust:

Creative Writing

Mon, 23 Feb 2026 13:31:06 +0000

My passion is to inspire people through writing. Most of it comes from being [[Creativity|creative]], but also [[Productivity|productive]], [[Creativity vs Productivity]].

Productive Procrastination

Mon, 23 Feb 2026 13:14:49 +0000

Productive procrastination is essential for our subconscious mind to process thoughts, and it is closely tied to [[Gut Feeling|Intuition]]. This perspective from a Reddit Comment resonates with me:

s3fs-fuse

Fri, 20 Feb 2026 16:49:20 +0000

s3fs allows Linux, macOS, and FreeBSD to mount an S3 bucket via FUSE(Filesystem in Userspace).
s3fs makes you operate files and directories in S3 bucket like a local file system.
s3fs preserves the native object format for files, allowing use of other tools like AWS CLI.

GitHub

AI won't Forget: A Problem?

Wed, 18 Feb 2026 16:39:26 +0000

With Claude code and other [[Generative AI]] trying to get as much data as possible—some of it is because data wins and they can know more about you—but also because it should help the AI to give you better answers.

New Ideas Daily

Wed, 18 Feb 2026 15:44:49 +0000

Similar to my [[Learn for Life]] mantra, a new idea daily. Do something simple within a day. It’s exhilarating.

Soren Iverson talks about it on Twitter.

S3 Storage Alternatives

Tue, 17 Feb 2026 15:53:10 +0000

[[AWS S3]] is the defacto standard for [[Storage Layer|object storage]]. Therefore there are many alternatives that implement the same API, but based on disk or other storages.

Modern OLAP Systems

Mon, 16 Feb 2026 22:16:19 +0000

With modern [[OLAP]] systems, you replace your [[Traditional OLAP Cubes]] one-to-one with another technology. Therefore, you keep everything the same on your current architecture but replace your cubes with a modern big data-optimized technology that focuses on the fastest query response time. See the Appendix for a comparison between modern OLAP Technologies.

RSS Feeds for Data Engineering

Mon, 16 Feb 2026 15:56:06 +0000

Besides the [[People of Data Engineering]], the most important is the content.

[!note] Newsboat

Recently I switched to [[RSS Readers]] called [[Newsboat]]. Check out the latest feeds directly in my dotfiles.

Newsboat: RSS-Feed for the Terminal

Mon, 16 Feb 2026 14:35:10 +0000

Newsboat is an [[RSS Readers|RSS/Atom feedreader]].

RSS and Atom are a number of widely-used XML formats to transmit, publish and syndicate articles, for example news or blog articles. Newsboat is designed to be used on text terminals on Unix or Unix-like systems such as GNU/Linux, FreeBSD or macOS.

Mermaid

Mon, 16 Feb 2026 10:39:00 +0000

JavaScript-based diagramming and charting tool that renders [[Markdown]]-inspired text definitions to create and modify diagrams dynamically.

Generate images like below with just text:
![[Mermaid.png]]

Change Data Capture (CDC)

Fri, 13 Feb 2026 10:06:05 +0000

Primary Purpose: It enables capturing and streaming changes made to the database. It’s used for integrating databases with other systems, such as replicating data to a data warehouse or syncing it with a cache or a search index.
Mechanism: CDC can be implemented using various techniques, including database triggers and polling, or by tapping into the database’s replication capabilities, such as using the WAL in PostgreSQL. It captures the “what” of the change (insert, update, delete) and the “how” (the actual data change).
Usage in Databases: It’s not a core database functionality for ensuring consistency or durability but is essential for modern data architectures that require real-time data integration and streaming.

See a practical example of [[Change Data Capture (CDC) - Real-Estate Project]].

Snapshotting

Thu, 12 Feb 2026 09:02:10 +0000

Contrary to [[Slowly Changing Dimension (Type 2)|SCD2]], which evolves over time, snapshotting captures the state of data at specific time intervals, much like [[Materialized Views]].

Data Modeling is Changing

Thu, 12 Feb 2026 08:17:29 +0000

Data Modeling is as much about [[Data Engineering Architecture]] as it is about modeling the data only. Therefore besides the below links, many approaches and common architecture you can find in [[Data Engineering Architecture#Common Architectures]].

Full Table Scan

Thu, 12 Feb 2026 08:03:54 +0000

A full table scan, also referred to as a sequential scan, involves reading every row in a [[Database]] table sequentially, evaluating each column against specified conditions. This approach is often the slowest method for scanning due to the substantial I/O reads it requires. These reads entail numerous seeks and expensive disk-to-memory transfers.

Import Files on Arch Linux (gphoto)

Tue, 10 Feb 2026 20:06:51 +0000

See gPhoto - ArchWiki, it is a [[Linux]] library to import images from a camera.

Find Camera

Plug in camera, e.g. canon -> use view mode and connect usb cable and get model:

Tools to Check Plagiarism & AI-Generated

Mon, 09 Feb 2026 13:18:39 +0000

Detect Plagiarism

[[Grammarly]] has had that feature now as well, see [[Grammarly#Check Plagiarism]]
Copyscape Plagiarism Checker (from [[Start Your Own Freelance Writing Business (Laura Briggs)]])
- there is a little pay if I want to check all: > Each search costs $0.03 (up to 200 words), plus $0.01 per extra 100 words.

Detect AI-Generated Articles

[[Originality.ai (detecting AI generated articles)]]

How to Detect AI Writing

[[How to detect AI Writing]]

Data Governance

Mon, 09 Feb 2026 09:47:17 +0000

Data governance encompasses a set of processes, roles, policies, standards, and metrics that are vital for the effective and efficient utilization of information in achieving an organization’s goals.

Git Worktree

Fri, 06 Feb 2026 17:17:14 +0000

Git worktree is an addition that comes automatically with [[git]] and can help you switching branches without the need of git stash and git checkout branch etc.

Rill Developer

Wed, 04 Feb 2026 19:49:01 +0000

Rill is a fast dashboards build trust and curiosity. Our fully-managed platform lets you focus on shaping data, not tuning databases and queries. Written in [[GoLang]].

Code Is Still the Best Abstraction

Wed, 04 Feb 2026 19:06:23 +0000

No-Code, or also called low-code is great for lowering the technical barriers and enabling more users to do data engineering. But these tools come with significant drawbacks:

Obsidian Smart Connections

Tue, 03 Feb 2026 13:40:11 +0000

Chat with your notes in [[Obsidian]]. Plus, see what’s most relevant in real time. Interact and stay organized. Powered by [[OpenAI]] [[ChatGPT]], GPT-4 & Embeddings.

Note Taking App / Tool

Thu, 29 Jan 2026 08:15:31 +0000

As of [[2022-10-14]]:
![[Pasted image 20221014142213.png]]

Tools:

[[Obsidian]]
[[Notion]]
[[Tana]]
[[OneNote]]
Logseq
Capacities
[[Roam]]
[[Evernote]]
[[Joplin]]
[[Google Keep]]
[[Milanote]]
[[Neovim]] // [[vim]] // Terminal solutions
- [[Brain- A Zettelkasten software for terminal users]]
- Zettelkasten with vim: GitHub - renerocksai/telekasten.nvimI
- zk: A plain text note-taking assistant
- [[VimWiki]] addon for managing notes
Thymer
Open Source
- French and German governments supported: suitenumerique/docs: A collaborative note taking, wiki and documentation platform that scales. Built with Django and React. Opensource alternative to Notion or Outline.

Origin: https://twitter.com/fortelabs/status/1579232715299205120/photo/1
References: [[Smart Note Taking]]

Social Media Scheduling Tools and Platforms

Tue, 27 Jan 2026 16:54:52 +0000

Used by [[Tiago Forte]]: Hypefury - Schedule & Automate Social Media Marketing
- Quite new and affordable compared to others.
- Hypefury - Schedule & Automate Social Media Marketing
HubSpot - Social Inbox
[[Buffer]] (free)
SmarterQueue - The Powerful All-In-One Social Media Scheduling Tool
- I used this once and found it excellent.
EvergreenFeed - Buffer queue on autopilot
- Enhances Buffer: EvergreenFeed (features calendar and bulk import)
Publer - Your Social Media Superhero
- Offers bulk import during the free trial and seems free for up to 5 profiles afterwards.
- the interface is not really good. hard to use
OneUp - Schedule your social media posts
- Supports bulk scheduling, only $4 a month, and looks promising.
[[Sprout Social]]
For [[Twitter|X/Twitter]] Threads specific
- Typefully - Twitter Thread Maker & Analytics
  - doesn’t have Bulk upload, but has an MCP that does this!
Fedica (Formerly Tweepsmap) (Origin: [[Chris Riccomini]])
[[Ordinal (Social Media)]]): Social Media Management Tool for Companies
PostWrite by [[Kevin Naughton Jr]]
Ferryman: It can sync your post from a master social media, different to others. Built by [[Kevin Naughton Jr]]

Open-Source

GitHub - gitroomhq/postiz-app: 📨 The ultimate social media scheduling tool, with a bunch of AI 🤖

API’s

Not a tool but APIs that can be integrated: [[Get Late]]

My Calendar in SmarterQueue: [[My Social Media]]. Some hashtags I used once: [[Hashtags social media]].

Data Lakehouse

Wed, 21 Jan 2026 18:48:19 +0000

A data lakehouse is an innovative data management architecture that seamlessly integrates the flexibility, cost-efficiency, and scalability of [[Data Lakes]] with the robust data management features and [[ACID Transactions]] typical of [[Data Warehouses]]. It leverages [[Data Lake Table Format]] to facilitate [[Business Intelligence]] and [[Machine Learning]] across comprehensive data sets.

Python

Wed, 21 Jan 2026 08:13:44 +0000

Python is a high-level, general-purpose programming language. Its design philosophy emphasizes code readability with the use of significant indentation. Python is dynamically-typed and garbage-collected. It supports multiple programming paradigms, including structured, object-oriented, and [[Functional Programming]].

Learning Data Engineering

Tue, 20 Jan 2026 09:38:17 +0000

There are many ways to learn data engineering. Besides the [[Books of Data Engineering]], below are different ways to learn.

People of Data Engineering

Tue, 20 Jan 2026 09:37:21 +0000

A list of the people of [[Data Engineering]] (sorted alphabetically). This list is not meant to be 100% complete, and descriptions are minimal. It’s a personal list that I am sharing with you.

Jetstream (AT Proto)

Tue, 20 Jan 2026 09:23:57 +0000

A simplified JSON event stream for [[AT Protocol]]

Good place to start is with the event stream docs: Event Stream - AT Protocol
and jetstream: GitHub - bluesky-social/jetstream: A simplified JSON event stream for AT Proto

E.g. [[Bluesky#^fb0c29]] are built on top of these Jetstreams.

Miso-glazed celeriac Steaks with Warm Asian Slaw

Sun, 18 Jan 2026 15:10:57 +0000

[!note] Disclaimer

This is AI-generated by [[DALL-E]] and ChatGPT when talking at the family table. [[Family Class - AI and ChatGPT]]. So not sure how good it actually tastes, I haven’t made it 😄

Docs for AI (`llms.txt`)

Fri, 09 Jan 2026 09:58:03 +0000

Writing for AI means making sure your docs are easy for AI to consume, as well as human users. The good news: Most discussions is that good docs for humans (plain language, well structured, semantic HTML) are also good docs for AI. [[Markdown]], and [[Static Site Generators (SSG)]] for the win.

Markup Language

Fri, 09 Jan 2026 09:29:45 +0000

A markup language is a set of rules that defines how the layout and presentation of text and images should appear in a digital document. It allows structuring documents, adding formatting, and specifying how different elements should be displayed (or “rendered”) on webpages.

SQLite DB

Tue, 06 Jan 2026 20:31:30 +0000

SQLite is a C-language library that implements a small, fast, self-contained, high-reliability, full-featured, SQL database engine. SQLite is the most used database engine in the world. SQLite is built into all mobile phones and most computers and comes bundled with countless other applications that people use every day. More Information…

Data Engineering Podcasts

Tue, 06 Jan 2026 18:57:20 +0000

Best Data Thoughtleader Interview Podcasts

[[The Data Engineering Podcast]] by [[Tobias Macey]] ( site, twitter, RSS)
The Analytics Engineering Podcast by [[dbt]] ( site, RSS)
The Data Stack Show by [[Rudderstack]] ( site, twitter, RSS)
Intricity101 Data Sharks Podcast ( site, twitter, RSS)
Drill to Detail with Mark Rittman ( site, twitter, RSS)

Markdown vs. Rich Text

Tue, 06 Jan 2026 06:35:40 +0000

[[Markdown]] is becoming the de facto standard for content writing. It has many excellent features such as [[Admonition (Call-outs)]], [[Mermaid]], and Drawings, all with essential text files.

SELECT Insights - Practical Data Engineering: A Hands-On Real-Estate Project Guide

Tue, 06 Jan 2026 06:35:40 +0000

Hello friends,

This week’s SELECT Topic is the updated GitHub repo I worked on many days to update from the ancient versions of dagster, delta lake, and MinIO to the latest.

Thoughts On Journal

Tue, 06 Jan 2026 06:35:40 +0000

Introduced by [[Derek Sivers]]. He used to journal for hours a day. He uses simple [[Plain Text Files]] and a handful of thoughts on files on different topics. Read his version on the Benefits of a daily diary and topic journals.

Data Lake File Formats

Mon, 05 Jan 2026 15:14:47 +0000

At the core of a [[Data Lake]] is the [[Storage Layer]], which predominantly houses files in open-source formats.

These data lake file formats are essentially the cloud’s version of CSVs but with enhanced capabilities. They are more column-oriented, facilitating efficient compression of large files and incorporating additional features. The leading formats in this space are Apache Parquet, Apache Avro, and Apache Arrow. They constitute the physical storage, with files distributed across various buckets within your storage layer.

Opinionated Data Platform (Closed-Source)

Mon, 05 Jan 2026 10:20:45 +0000

An opinionated data platform is a platform that has choosen tools from the [[Open Data Stack]], and integrated them to a unified platform. This closed-source data platform works out of the box immediately.

vim

Sun, 04 Jan 2026 18:09:09 +0000

Vim is a text-based editor that is running in the terminal.

Unlike other editors, Vim’s power is its [[Vim Language (and Motions)]]. Although this can be added as an extension in other editors, vim has much more to offer.

Connecting The Dots

Sat, 03 Jan 2026 20:35:36 +0000

A saying that [[Steve Jobs]] said in his commencement speech:

You can’t connect the dots looking forward; you can only connect them looking backward. So, you have to trust that the dots will somehow connect in your future.

Vanity Metric

Sat, 20 Dec 2025 06:57:29 +0000

[[Social Media]] is just a vanity metric.

Posted on Bsky: Followers and likes are just vanity numbers. Having a purpose and clearly defined principles is all that counts. Great read, could be my words.

Apache Kafka

Tue, 16 Dec 2025 23:44:27 +0000

Apache Kafka is an open-source distributed event [[streaming platform]] used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.

Future of Web

Fri, 05 Dec 2025 09:40:55 +0000

The future of the web might be around the corner, back to the roots with HTML pages:
![[Screenshot_20241030-071955~2.png]]

[[Bluesky]] with their [[AT Protocol]], changing the way back to an open web from social media giants:
![[Screenshot_20241030-072005~2.png]]

Kaggle

Wed, 03 Dec 2025 10:06:34 +0000

Kaggle is the world’s largest [[Data Science]] community with powerful tools and resources to help you achieve your data science goals.

Markdown Presentations or Slides

Wed, 03 Dec 2025 07:07:11 +0000

Creating slides on Markdown.

Slide Tools

[[Marp - Presentation Markdown]]
[[Slidev]]
lookatme (terminal)
[[Quarto]]
reveal.js used by [[Letsboot.ch]]
[[Presenterm]]: A markdown terminal slideshow tool link used by No boilerplate
WebSlides: Create Beautiful HTML Presentations
iA Presenter (paid, apple only)

Presenterm

Wed, 03 Dec 2025 07:06:32 +0000

Presenterm, a terminal [[Markdown Presentations or Slides]] tool, written in Rust.

Presenterm is just goated. Quickly set your color theme and write some Markdown, and it looks gorgeous!

Data Mesh: Decentralizing and Unifying Data Resources

Tue, 02 Dec 2025 16:50:52 +0000

In the evolving landscape of data management, Data Meshes emerge as a crucial concept, trying to bridge the gaps between isolated data teams. Their core value lies in fostering a shared understanding and usage of data across diverse teams within an organization.

AWS S3

Tue, 02 Dec 2025 13:37:14 +0000

Amazon Simple Storage Service (Amazon S3) is a highly scalable, secure, and durable object storage service provided by Amazon Web Services (AWS).

Storage Layer (Object-Store)

Tue, 02 Dec 2025 13:36:20 +0000

The Storage Layer, encompassing object storage services from the three major cloud providers - [[AWS S3]], [[Azure Blob Storage]], and [[Google Cloud Storage]] - is an integral part of modern data infrastructure. This layer supports [[Data Lake File Formats]] and [[Data Lake Table Format]], offering a robust foundation for data management.

Boredom

Mon, 01 Dec 2025 18:57:40 +0000

Boredom isn’t a bug, it’s a feature. Harvard professor [[Arthur Brooks]] explains why boredom unlocks [[creativity]], activates a powerful brain network, and might even protect you from [[depression]].

Ibis (Python)

Sun, 30 Nov 2025 07:42:48 +0000

Ibis is a [[Python]] library that provides a lightweight, universal interface for [[data wrangling]]. It helps Python users explore and transform data of any size, stored anywhere.

Export YouTube Subscriptions as RSS Feeds (Python)

Fri, 28 Nov 2025 08:36:01 +0000

For my [[My RSS Bookmarks]] and [[RSS feeds for Data Engineering]] I wanted to export all my Subscriptions to add to [[Newsboat]], for that I need this format:

Open Data Stack

Thu, 27 Nov 2025 11:58:57 +0000

As data engineering evolves, it’s crucial to understand the potential of open-source solutions that adhere to open standards, providing an integral and adaptable framework known as the Open Data Stack. This approach outshines the term ‘[[Modern Data Stack]],’ which has garnered some negativity and created confusion within the industry. So, what exactly is the Open Data Stack, and why should we focus on it?

SELECT Insights - What the Heck Is the Open Data Stack?

Thu, 27 Nov 2025 11:58:57 +0000

Hello friends,

This week’s SELECT Topic is the Open Data Stack. This is something I have written before, but I’d like to share my view on the differences between the Modern Data Stack and what I see behind the term.

My Gigs (Podcasts, Videos, Slides, etc.)

Wed, 26 Nov 2025 13:23:52 +0000

Here I have a list of my appearance online or slides I shared during an event.

10. Knowledge Management in the Digital Age: From Zettelkasten to Startup Owner

It was part of Swiss Digital Days 2025 organized amoung [[Berner Fachhochschule]] and [[Stadtbibliothek Biel]]. I translated the talk and recorded it to upload to YouTube.

Different Types of Writers

Wed, 26 Nov 2025 12:39:33 +0000

Here I collect the different types of writer I came across so far.

Two Types of Writers

There are at least two types of writers:

History of General Architecture in Data

Tue, 25 Nov 2025 10:34:42 +0000

KEYNOTE: [[Hannes Mühleisen]] - Data Architecture Turned Upside Down at the PyData in Amsterdam 2025.

Below some of my prelimitary notes, and also note, that this is heavily focused on [[DuckDB]]. But still has it’s relevance and mostly true with the [[Small Data Manifesto]], and small data revolution.

OLTP (Online Transactional Processing)

Mon, 24 Nov 2025 16:32:18 +0000

OLTP stands at the core of daily digital transactions, powering systems like ATMs, online banking, cash registers, and e-commerce platforms. It represents the [[Operational Database]] or the database of an application, typically structured using a [[Relational Model]].

Data Vault

Mon, 24 Nov 2025 11:05:43 +0000

A data vault is a [[Data Modeling]] approach used to build a data warehouse for enterprise-scale analytics. The data vault has three types of entities: hubs, links, and satellites.

dockerfile

Fri, 21 Nov 2025 12:32:20 +0000

A dockerfile can be used to build an image and then run as a [[container]] on docker terms or as a pod in [[Kubernetes]].

Personal Knowledge Management (PKM)

Wed, 19 Nov 2025 19:19:54 +0000

The PKM workflow to capture knowledge and todos is essential for any human, developer, or knowledge worker. It should reduce stress and improve your quality of life in a way you wouldn’t believe before you had one.

Data Lake

Mon, 17 Nov 2025 07:33:05 +0000

A Data Lake is a versatile storage system, found within the [[Storage Layer]], containing a vast array of both unstructured and structured data. This data is stored without a predetermined purpose, allowing for flexibility and scalability. Data Lakes can be built using a variety of technologies, including Hadoop, NoSQL, Amazon Simple Storage Service, and relational databases, and they accommodate diverse data formats such as Excel, CSV, Text, Logs, and more.

Recursive Search in Terminal with fzf (`ctrl+r`)

Fri, 14 Nov 2025 11:15:06 +0000

The videos I got it from:

Installation

GitHub - junegunn/fzf: :cherry_blossom: A command-line fuzzy finder

ETL Tools or Platforms

Fri, 14 Nov 2025 10:36:53 +0000

Older Generation (Traditional ETL Tools):

[[SSIS]] (SQL Server Integration Services)
[[Informatica PowerCenter]]
[[Oracle Warehouse Builder (OWB)]] later evolved into [[Oracle Data Integrator (ODI)]]
[[IBM DataStage]]
[[SAP Data Services]]
[[Talend Open Studio for Data Integration]] (open-source)
[[Pentaho Data Integration]] (open-source)
[[TinyETL]]: Fast, zero-config ETL in a single binary

Newer Generation (Modern ETL/[[ELT Tools]]):

[[Apache Airflow]] (open-source)
[[Airbyte]] (open-source)
[[StitchData|Stitch]] (closed-source)
[[Fivetran]] (closed-source)
[[Matillion]] (closed-source)
[[AWS Glue]] (closed-source)
[[Google Cloud Dataflow]] (closed-source)
[[Apache NiFi]] (open-source)
[[StreamSets]] (open-source)
[[Meltano]] (open-source)

This list includes a mix of both open-source and closed-source solutions, illustrating the evolution and diversity in the ETL/ELT landscape.

Why having a Data Warehouse

Fri, 14 Nov 2025 10:05:22 +0000

Besides the obvious reasons of a shop explained above, a [[data warehouse]] gives you big advantages:

In theory: Integration and transformation of raw data of an organisation from multiple sources (mostly very structured like SAP, CRM, Excel, etc.) into meaningful and useful information, historical stored.
In practice: Similar a cockpit in an aeroplane - All Measures and [[KPI|KPIs]] are at one place in order to steer the plane and take the right decisions
It allows businesses to make better decisions by accessing the data well structured
- It helps to make informed decisions based on facts
- It improves the efficiency of business operations
- It enhances the quality of customer service
The visual products from business intelligence based on top of a data warehouses are largely:
- Ad hoc Reporting
- Standard Reporting
- Dashboards / Cockpits

Origin: Data Engineering, the future of Data Warehousing?
References: [[What is a Data Warehouse]]

What is a Data Warehouse

Fri, 14 Nov 2025 10:04:59 +0000

What is data warehousing or what is a business intelligence engineer doing, and why are they using a [[data warehouse]]?

ELT (Extract, Load, Transform)

Fri, 14 Nov 2025 10:04:10 +0000

ELT (Extract, Load, and Transform) represents a [[Data Integration]] methodology where data is first extracted (E) from source systems, then loaded (L) as raw data into a target system, followed by transformation (T) within the target. This approach, executed within the destination [[Data Warehouse]], contrasts with the traditional [[ETL]] method, where data undergoes transformation prior to reaching its destination. For a comprehensive comparison, see [[ETL vs ELT]].

Inmon vs Kimball: Data Warehousing Approaches

Fri, 14 Nov 2025 09:52:39 +0000

The comparison on the ongoing debate between [[Bill Inmon]] and [[Ralph Kimball]].

Comparison

Inmon (Bill Inmon):

Emphasizes a highly normalized structure. Begins directly with the Core, which remains normalized and closely aligned with the source system.
Denormalization occurs in the [[Data Mart]]s, adhering more to a [[Snowflake Schema]].

Kimball (Ralph Kimball):

Initiates with a staging area, where the Core is already denormalized.
- Here, the data mart effectively becomes the data warehouse.
Known as a bottom-up approach, it focuses on creating a [[Star Schema]].
Creator of [[Dimensional Modeling]].

Although both methodologies have nuanced differences, they ultimately share many similarities. Personally, I lean towards Kimball’s approach. It allows for staging data and then normalizing from that point, avoiding the replication of the [[OLTP]] source system’s structure beyond the staging phase.

Data Warehouse Automation (DWA)

Fri, 14 Nov 2025 09:37:39 +0000

A very long time ago, 1986, 31 years ago to be precise, IBM in Europe created the very first architecture of a [[data warehouse]]. And it seems to be a masterpiece as it hasn’t changed much since. How can we improve or bring some innovation into the Data Warehouse business in times when everyone is talking about big data, [[Data Lake]]s, the Internet of Things (IoT), predictive analytics, Data Vault, etc.?

Data Modeling Tools

Thu, 13 Nov 2025 22:21:42 +0000

Popular data modeling tools include Sqldbm, DBDiagrams, Enterprise Architect, and SAP PowerDesigner. These tools are widely used in the industry and offer powerful features such as data modeling, profiling, and visualization.

Neovim set-up like VS Code

Thu, 13 Nov 2025 09:42:06 +0000

To get started with [[Neovim]], I collected a list of these plugins that helped me replicate a [[VSCode]]-like experience

APZelos/blamer.nvim: A git blame plugin for Neovim inspired by VS Code’s GitLens plugin
sindrets/diffview.nvim: Single tab page interface for easily cycling through diffs for all modified files for any git rev.
nvim-neo-tree/neo-tree.nvim: Neovim plugin to manage the file system and other tree-like structures.
stevearc/aerial.nvim: Nice code outline window to jump to representative methods or function
neovim/nvim-lspconfig: Quickstart configs for Nvim LSP (Language Server Protocol). Which provides language features like auto complete, go to definition, find all references, etc.
akinsho/bufferline.nvim: Nice tabs and buffer handling with lots of extra features.
tpope/vim-commentary: A superior commenting experience across all programming languages.
nvim-telescope/telescope.nvim: Elegant fuzzy finder to find, filter, preview, and pick with 100 powerful features that benefit every workflow.
Search and replace: Developing Search & Replace (similar to VS Code) with nvim-spectre : r/neovim. See also [[Advanced search and replace (like VSCode) in Neovim (cdo)]].

And if you’re feeling unsure about making the switch, don’t worry, this topic is definitely more on the nerdy side of things 😜. But who knows, you might discover a new toy and find it a fun exploration writing text or code!

Wikipedia Data sets

Wed, 12 Nov 2025 07:38:54 +0000

from Wikimedia Downloads: Analytics, Pageviews: statistics compiled using the current Pageview Definition. Available as:

Pageview complete: Our best effort to provide a comprehensive timeseries of per-article pageview data for Wikimedia projects. Data spans from December 2007 to the present with a uniform format and compression.
Pageview/projectview data filtered to what we believe is only human traffic. Available since May 2015.
Pageview/projectview data, highly compressed and corrected for outages. This dataset was historically computed using the best source available at the time:
- 2015 - 2025: Index of /other/pageviews/
  - Dec 2015 - 2022: https://dumps.wikimedia.your.org/other/pageviews/
    - compressing and correcting: pageviews dataset
- 2007 - Dec 2015: compressing and correcting: pagecounts-raw dataset
  - 2007 to 2011: from pagecounts-raw (to be loaded during the second half of October 2020)
  - 2011 to 2015: from pagecounts-ez

Others

Wikipedia Dumps: Format: XML, SQL dumps
- Link to ZIP: Index of /other/pageviews/
- Content: Full page content, revision history, user data, and more
- Updated: Typically twice monthly
- most commanly used:
  - pages-articles.xml.bz2 - Current versions of article content (most popular for analysis)
  - pages-meta-history.xml.bz2 - Full revision history (very large)
  - page.sql.gz - Page metadata tables
  - categorylinks.sql.gz - Category relationships
- Pre-processed Datasets
  - Wikipedia Corpus on Hugging Face - Cleaned text datasets ready for ML/NLP
  - DBpedia ( https://www.dbpedia.org/) - Structured data extracted from Wikipedia infoboxes
  - Wikidata ( https://www.wikidata.org/) - Structured knowledge base with JSON dumps
API: Load MediaWiki data in Python using dltHub

Downloading

Querying with DuckDB

-- This is the actual public data format
describe
FROM read_csv('https://dumps.wikimedia.org/other/pageviews/2025/2025-10/pageviews-20251001-000000.gz',
    delim=' ',
    header=false,
    compression='gzip')
LIMIT 10;


--auto detection:
SELECT *
FROM read_csv(
    'https://dumps.wikimedia.org/other/pageviews/2025/2025-10/pageviews-20251001-000000.gz',
    delim=' ',
    header=false,
    compression='gzip',
    all_varchar=true  -- Initially read all as text to see the raw data
)
LIMIT 20;

--manually
-- When using the columns parameter, use the names you defined
SELECT
    project,
    page_title,
    view_count,
    bytes_served
FROM read_csv(
    'https://dumps.wikimedia.org/other/pageviews/2025/2025-10/pageviews-20251001-000000.gz',
    delim=' ',
    header=false,
    compression='gzip',
    columns={
        'project': 'VARCHAR',
        'page_title': 'VARCHAR',
        'view_count': 'BIGINT',
        'bytes_served': 'BIGINT'
    }
)
WHERE project IS NOT NULL
LIMIT 10;


---predicate pushdown

SELECT
    page_title,
    view_count,
    bytes_served
FROM read_csv(
    'https://dumps.wikimedia.org/other/pageviews/2025/2025-10/pageviews-20251001-000000.gz',
    delim=' ',
    header=false,
    compression='gzip',
    columns={
        'project': 'VARCHAR',
        'page_title': 'VARCHAR',
        'view_count': 'BIGINT',
        'bytes_served': 'BIGINT'
    }
)
WHERE page_title IN (
    'DuckDB',
    'Apache_Spark',
    'Apache_Kafka',
    'Data_engineering',
    'Extract,_transform,_load',
    'ClickHouse',
    'Snowflake_(company)',
    'Apache_Airflow',
    'Databricks',
    'Data_warehouse',
    'Data_lake',
    'dbt_(software)',
    'Apache_Flink',
    'PostgreSQL',
    'BigQuery'
)
LIMIT 1000;

ClickHouse way

From WikiStat | ClickHouse Docs:

Traditional OLAP Cubes Replacements

Tue, 11 Nov 2025 08:59:52 +0000

Because you can make a one-to-one replacement, have a faster cloud backend, virtualize a [[Semantic Layer]], or use a service cloud provider, I categorized the different technologies into the following groups

Why replace Traditional OLAP Cubes?

Tue, 11 Nov 2025 08:58:24 +0000

In my job as a business intelligence specialist and data engineer, I was always using [[OLAP]]-Technologies for quick ad-hoc analysis with tools like Tableau, TARGIT, Power BI, [[Excel]], [[BI Tools|etc.]] As the Microsoft BI stack is still used widely in small to large companies, the technology for it is mostly SQL Server Analysis Services ([[SSAS]]) or Microsoft Cubes.

AWS S3 SDK Credential Auto-Detection Issue with Public Buckets (ClickHouse)

Thu, 06 Nov 2025 14:09:44 +0000

This was found in realation with [[ClickHouse]] example on clickhouse-modeling-rill-example/sources/noaa-weather.yaml.

Problem

ClickHouse fails to read from public S3 buckets with error:

The Data Engineering Toolkit

Thu, 06 Nov 2025 11:03:35 +0000

The Data Engineering Toolkit

To thrive as a data engineer, you need various skills—from fundamental (Linux commands, containerization, programming languages) to Kubernetes orchestration. The data engineering toolkit provides the building blocks of data engineering work in 2025.

Compute and Storage Separation

Tue, 04 Nov 2025 08:40:57 +0000

The compute-storage separation model has become increasingly popular in cloud OLAP systems because it allows for independent scaling of resources and often supports features like storage-based billing and compute-based billing separately.

Apache Druid

Tue, 04 Nov 2025 08:17:13 +0000

Druid is an open-source, column-oriented, distributed data store written in Java. It’s designed to quickly ingest massive quantities of event data, and provide low-latency queries on top of the data.

Data Engineering Architecture

Tue, 04 Nov 2025 08:11:15 +0000

A good data engineering architecture is hard.

But the best one IMO is the overview from A15z on Emerging Architectures for Modern Data Infrastructure ([[Emerging Architectures for Modern Data Infrastructure (a16z)|int. Link]] or Zotero).

Iceberg Table

Mon, 03 Nov 2025 11:52:47 +0000

Iceberg Tables, an innovation by [[Snowflake]] built upon [[Apache Iceberg]], itself rooted in [[Apache Parquet]], are now available in a private preview. This significant feature has been launched across all three major cloud platforms simultaneously! You can contact your account team to get started today. Source

Data Contracts

Thu, 30 Oct 2025 20:14:47 +0000

[!warning] WIP: This is an unfinished Note

Below are my uncleaned notes, I shared them as these are hopefully helpful to others. Once I have more time, I will convert it into a nice blog or article. For now, it’s a bit messy.

CDP - Customer Data Platform

Mon, 27 Oct 2025 14:53:44 +0000

A customer [[Data Platform]] (CDP) is a system that collects large quantities of customer data from a variety of channels and devices, enhancing accessibility for those who need it. CDPs sort, categorize, and cleanse data, removing inaccuracies or outdated information.

Document Signer

Sat, 25 Oct 2025 18:24:24 +0000

Signing a document for a contract, or similar. The OG is probably [[DocuSign]]. Below you see some common tools.

Tools

[[DocuSign]]: OG.
[[Documenso]]: Host yourself and get everything for free, as it is open-source. If you don’t want to self-host, you get 5 for free and unlimited when paying 25$ per month.
[[BoldSign]] ( link): Includes 25 signature requests/month. If you pay 120
[[SmallPDF]]: Sign PDF | eSign Your PDFs with Digital Signatures Online
[[Zoho Sign]]: How to digitally sign a PDF online | Zoho Sign
Google Drive has also one.
Swiss electronic signatures. Not Swiss prices. | DeepSign - connected with [[AbaNinja]]
- DeepSign by DeepCloud

Prices

DocuSign is expensive. For 120$ a year, you get only 5 signs a month. Apparently, BoldSign includes 25 signature requests/month.

YAML

Fri, 17 Oct 2025 09:38:14 +0000

YAML is a data serialization language that is often used for writing configuration files. Depending on whom you ask, YAML stands for yet another markup language or YAML ain’t markup language (a recursive acronym), which emphasizes that YAML is for data, not documents.

Pivot Table

Thu, 16 Oct 2025 18:00:01 +0000

Are pivot tables making a comeback in 2024? [[Rill Developer]] incorporates pivot tables within its web UI, [[Cube]] enables pivot table functionality in Excel WITH Cube, and [[DuckDB]] has introduced its own implementation. Tweet

Liquibase (Track DB Changes)

Thu, 16 Oct 2025 13:34:55 +0000

Liquibase is a database [[Schema Evolution|schema change management]] tool that helps track, version, and deploy database changes - particularly useful for changing database schemas over time. It works similar to version control systems (like [[Git]]), but specifically for database changes (and [[Schema Evolution]]).

Managing My Business with Obsidian (CRM)

Thu, 16 Oct 2025 12:49:04 +0000

I manage my business mostly with [[Obsidian]]. I have a note for each company (I add Inc to each note name “Company Inc”) and add them to my CRM. Inc is helpful because I usually have another note for the company itself or if they have a product, so I have a clear separation between my “notes” and part of the second brain and business-related CRM notes.

Data Engineering Communities

Tue, 14 Oct 2025 11:19:38 +0000

Communities for data engineering.

Modern Data Community by [[Michael Kahan]] (not that active anymore)
The Technical Freelancer Academy by [[Ben Rogojan|SeattleDataGuy]]. This community is about Technical Freelancing in the data space, but as lots of data engineers are there, it’s a great community and super active. There’s also a newsletter.
Practical Data Engineering Discord by [[Joe Reis]]. Super active and welcoming community. Find info on this Notion page ( Contribute, Events and more)

Slack Communities

Some great communities live in Slack and have many members, here are some of them. Member counts are as of [[2025-07-10]].

Relational Model

Mon, 13 Oct 2025 13:35:55 +0000

The relational model is an approach to managing data using a structure and language consistent with first-order predicate logic, first described in 1969 by English computer scientist [[Edgar F. Codd]], where all data is represented in terms of tuples, and grouped into relations. His paper about a Relational Model of Data for Large Shared Data Banks.

Using Rill MCP

Mon, 13 Oct 2025 11:11:24 +0000

Trying [[Rill Developer|Rill]] MCP implementation with [[Claude Desktop]]. It worked on the first try with a local Rill and NYC and Pickup data set.

OLAP (Online Analytical Processing)

Fri, 10 Oct 2025 10:04:30 +0000

What is OLAP?

[[OLAP]] stands for Online Analytical Processing. It’s a powerful tool for multidimensional business data analysis, offering complex calculations, trend analysis, and advanced data modeling capabilities.

Writing Effectively (with Value)

Thu, 09 Oct 2025 06:20:14 +0000

Write valuable things to your readers! Don’t explain as in school. Only create new/original knowledge (if it’s useful, who cares?).

Hell Yeah or No: The Power of Saying No

Wed, 08 Oct 2025 10:59:00 +0000

A lot about life and work is about saying no. The mental model of “ Hell Yeah, or No” served me well to focus on things that giving me energy. If many things make you happy, know you can only do so much.

Screen Sharing on Wayland (Hyprland) with Chromium Browsers (Brave, Chrome, Chromium, etc.)

Tue, 07 Oct 2025 20:20:14 +0000

This is the solution to avoid double prompt when screen sharing on Linux with Wayland. This works for Google Meet, Zoom, Teams and other that run in the browser.

Pick your Battles

Fri, 03 Oct 2025 08:16:27 +0000

When writing, choose your battles. You can’t write about everything. Avoid traps from the get-go, either because you don’t like it or don’t believe in. The same goes for life.

Change Data Capture (CDC) - Real-Estate Project

Thu, 02 Oct 2025 06:43:22 +0000

CDC is a powerful tool and especially in cloud environments with event-driven architectures. I used it to minimize the downloads of already downloaded properties. Besides existing open-source CDC solutions like Debezium, I implemented my own simple logic to detect the changes. Also because I have no access to the source-OLTP database where the properties are stored which you’d need.

Nimble

Wed, 01 Oct 2025 20:32:27 +0000

New [[Data Lake File Formats|File Format]] for storage of large columnar datasets.

Nimble (formerly known as “Alpha”) is a new columnar file format for large datasets created by Meta. Nimble is meant to be a replacement for file formats such as Apache Parquet and ORC.

Adding Printer on Linux

Mon, 29 Sep 2025 13:24:07 +0000

This example are for my [[Brother DCPL2550DN]] currently having

UI

system-config-printer - CUPS configuration tool.

![[img_Adding Printer on Linux_1755711511690.webp]]

Drivers are installed

check what

DSL (Domain Specific Language)

Thu, 25 Sep 2025 07:52:59 +0000

A Domain-Specific Language (DSL) is a computer language targeted at a particular kind of problem, rather than a general-purpose language aimed at any kind of software problem.

Devcontainers (`devcontainer.json`)

Wed, 24 Sep 2025 11:21:31 +0000

The [[VSCode|Visual Studio Code]] Dev Containers extension lets you use a Docker container as a full-featured development environment. It allows you to open any folder or repository inside a container and take advantage of Visual Studio Code’s full feature set.

IDE (Integrated Development Environment)

Wed, 24 Sep 2025 11:21:31 +0000

IDEs are Developer Environments such as [[VSCode]], [[PyCharm]], [[vim]], etc.

An alternative term that comes from the [[Neovim]] community is called [[PDE (Personalized Development Environment)]].

Microservices (Architecture)

Mon, 22 Sep 2025 21:27:01 +0000

The core concepts of microservice architecture are:

Definition: Microservices are an architectural style where applications are structured as a collection of loosely coupled, independently deployable services.
Key Architectural Layers:
- Presentation Layer: Handles user interface and client interactions
- Business Process Layer: Contains core business logic and workflows
- Data Access Layer: Manages data storage and retrieval
Communication Patterns:
- Direct Client-to-Service: Simple point-to-point communication
- [[Service-Oriented Architecture]]: Centralized message routing through an [[Enterprise Service Bus]]
- API Gateway/Proxy: Modern approach with distributed service mesh

Architecture Overviews

Helpful images from [[Microservices Udemy Course]].

Vault (HashiCorp)

Mon, 22 Sep 2025 21:16:30 +0000

A tool for secrets management, encryption as a service, and privileged access management, created by [[HashiCorp]].

Integration with Postgres and Jupyter Noteboo

An example of [[Jupyter Notebook]] integration with Postgres, that requests a new secreat from Vault:

Docker

Sun, 21 Sep 2025 15:27:03 +0000

Docker is a set of platform-as-a-service products that use OS-level virtualization to deliver software in packages called containers.

The service has both free and premium tiers. The software that hosts the containers is called Docker Engine. It was first released in 2013 and was developed by Docker, Inc.

Check Disk Size

Sat, 20 Sep 2025 11:21:28 +0000

If the disk is full, annoyingly, what do you do? You search for a tool to check sizes and details.

stu

Fri, 19 Sep 2025 11:30:03 +0000

GitHub - lusingander/stu: TUI explorer application for Amazon S3 (AWS S3) 🪣

STU, [[AWS S3|S3]] Terminal UI, is a interactive terminal-based explorer for Amazon S3 (AWS S3).

Unix Philosophy

Fri, 19 Sep 2025 09:38:23 +0000

Here are some examples:

Small is beautiful.
Make each program do one thing well.
Build a prototype as soon as possible.

[!abstract] Abstract

Semantic Layer Measure Defininition Examples

Thu, 18 Sep 2025 14:10:16 +0000

Example from [[Cube]] with SQL Expressions:

cubes:
  - name: orders
    sql_table: orders
 
    measures:
      - name: statuses
        sql: "STRING_AGG(status)"
        type: string
 
    dimensions:
      - name: status
        sql: "UPPER(status)"
        type: string

Or more specifically designing [[Metrics]]:

Todo Apps

Thu, 18 Sep 2025 12:13:13 +0000

[[NirvanaHQ]]
[[Todoist]], [[Wunderlist]], [[Microsoft TODO]], google, [[TickTick]], the blue todo
TeuxDeux: The to-do list app for organizing everything used by [[Matt D’Avella]]

Futher Reads

[[Project Management Tools]]
[[Managing ToDos Personally (ToDo App)]]
I Tried Every Todo App and Ended Up With a .txt File - Alireza Bashiri

Origin: [[NirvanaHQ]]
References: [[Managing ToDos Personally (ToDo App)]]

Project Management Tools

Thu, 18 Sep 2025 12:10:46 +0000

My Favorites

[[Basecamp]] with [[Shape Up]] methodology
[[Managing Projects with Markdown]]
[[Fizzy]] by [[37signals]]: Let’s do another Fizzy demo! We should all know by now that the longer… | Jason Fried | 19 comments

CLI

zeit: Zeit, erfassen. A command line tool for tracking time spent on activities.
[[Calcure]]: Terminal Calendar

Most Popular

Popular project management tools similar to [[Basecamp]]:

Getting Things Done (GTD)

Thu, 18 Sep 2025 12:07:59 +0000

Up until now, we managed ideas and knowledge, which enlarges the [[Second Brain]] and how to link our thoughts. Besides this, much anxiety and stress come from not being on top of your tasks, jobs, things you might want to do, ideas, and lists you want to read. The to-dos you put yourself somewhere on a post-it, a todo app, or anywhere else. I also tried to incorporate to-dos in my note-taking apps but never succeeded. It works best for me when having a dedicated tool and workflow.

Daydream

Wed, 17 Sep 2025 07:38:14 +0000

It’s not as bad as we thought, [[Johann Hari]] says in his book [[Stolen Focus- Why You Can’t Pay Attention (Johann Hari)]]. Only if we are relaxed and we have time to daydream, our ideas, our [[Creativity]] is flourishing. ^7fca87

VertiPaq (xVelocity)

Tue, 16 Sep 2025 14:49:54 +0000

The VertiPaq engine is an in-memory [[Columnar]] database powering [[Excel|MS Excel]] [[Power Pivot (Excel)|Power Pivot]], SQL Server Analysis Services ([[SSAS]]) Tabular, and [[Power BI]]. When data is loaded into a data model, the VertiPaq engine compresses and stores it in RAM for efficient processing.

What is an Analytics API?

Mon, 15 Sep 2025 10:43:18 +0000

Now, as we’ve seen what GraphQL can do, we talk about building an API that leads to the next level of data engineering, which I call Analytics API in this article. The API will empower all stakeholders to use one single source of accessing analytics data in a consistent and decoupled semantic way (and if you know a better name, please let me know!).

Zettelkasten

Fri, 12 Sep 2025 14:13:55 +0000

Started by [[Niklas Luhmann]], also called “Slip Box”. This approach will further dilute the idea of a librarian who organizes books in main categories and sub-folder and focuses only on the thought at hand as an individual note. Instead of spending lots of your brainpower on which folder to put a specific note in, you start creating it in one folder for all and linking them together. Usually, you wouldn’t have any structure in these concepts, but I combined the two methods into my Second Brain. You also see in the above design under 💡 Resources/🗃 Zettelkasten I have its folder.

Medallion Architecture

Fri, 12 Sep 2025 08:47:42 +0000

The databricks medallion architecture is not really an architecture, but more [[Data Modeling#Different Levels|Approach or Pattern]] with three data stages: bronze, silver, and gold.

Datasette

Thu, 04 Sep 2025 08:52:22 +0000

An open source multi-tool for exploring and publishing data

Datasette is a tool for exploring and publishing data. It helps people take data of any shape or size and publish that as an interactive, explorable website and accompanying API.

Dogsheep

Thu, 04 Sep 2025 08:48:23 +0000

Tries to create a [[Personal Data Warehouse]] with open-source SQLite and sorts. You can load Twitter, google Drive, and sort into a local database.

Change Color Scheme Dark Reader Extension

Fri, 29 Aug 2025 06:10:18 +0000

Chrome Web Store - Extensions

Sometimes it’s hard to find the Pre-defined schemes. But they are here:

![[Pasted image 20220829153324.png]]
and then
![[Pasted image 20220829153339.png]]

Marketing

Thu, 28 Aug 2025 21:59:48 +0000

Marketing is a way of transporting enthusiasm. [[Jason Fried]], more

It can be either good or bad [[enthusiasm]], but usually, it is contagious & refreshing. Curated or manufactured (fake) enthusiasm no longer works well. Real enthusiasm is coming back. Authenticity, genuine, especially with [[Generative AI|AI]]..

Flash my KBD Fans Keyboard

Tue, 26 Aug 2025 11:44:58 +0000

QMK Tutorial: QMK Toolbox (Flashing Firmware On Your Keyboard)

Set-up / Flash

New PCB DZ65RGB V#

[[2025-04-11]]

[!warning] MacOS: The web version could not save the JSON

ORM (Object-Relational Mapping)

Mon, 18 Aug 2025 08:27:04 +0000

ORM in computer science is a programming technique for converting data between a [[Relational Databases|relational database]] and the heap of an [[Object-Oriented Programming]] language.

Free Images/Pictures for Content (Stock Images)

Mon, 18 Aug 2025 06:30:29 +0000

Find the right pictures.

Free:

https://www.pexels.com - Free Stock Photo & Video License - Pexels
- What is allowed? 👌
  - All photos and videos on Pexels are free to use.
  - Attribution is not required. Giving credit to the photographer or Pexels is not necessary but always appreciated.
  - You can modify the photos and videos from Pexels. Be creative and edit them as you like.
- What is not allowed? 👎
  - Identifiable people may not appear in a bad light or in a way that is offensive.
  - Don’t sell unaltered copies of a photo or video, e.g. as a poster, print or on a physical product without modifying it first.
  - Don’t imply endorsement of your product by people or brands on the imagery.
  - Don’t redistribute or sell the photos and videos on other stock photo or wallpaper platforms.
https://unsplash.com
Freepik | Create great designs, faster used by [[David Heinemeier Hansson (DHH)|DHH]] for [[Omakub]]

Payed:

www.shutterstock.com

Editing

[[Canva]] - Canva: Visual Suite for Everyone

[[AI Generated Images]]

AI-powered stock photos | StockAI
Profile image
- Avatar AI by [[levelsio]] - Create Your AI Avatars, AI Profile Pictures, AI Photography, Pet Portraits, Dating Photos, Couples Photoshoot, LinkedIn Professional Profile Photos with Artificial Intelligence using Stable Diffusion and Dreambooth
- AI Profile Picture Generator | For humans, dogs and cats
Interior AI Free - Interior Design Ideas and Virtual Staging App using Artifical IntelligenceA
[[Mermaid]] with mermaidchart.com
[[SVG]] generation with Infinite AI Artboard - Recraft

Related to [[Generative AI]].

Modern Semantic Layer

Thu, 14 Aug 2025 14:00:42 +0000

I have written about The Rise of the Semantic Layer, where I stated that a [[Semantic Layer]] is nothing new; it has existed within [[BI Tools]] since its inception.

Logical Data Model

Thu, 14 Aug 2025 13:53:53 +0000

Creating a logical data model to represent the business requirements and a conceptual model to provide a high-level view of the data. The [[Physical Data Model|bottom-up]] approach is beneficial when dealing with legacy systems or when there needs to be more knowledge of the organization’s data requirements.

Types of Content (Online)

Sun, 03 Aug 2025 07:57:31 +0000

I think there are main types of content and it’s sub categories.

[[Books]]: Extended works on specific subjects, typically with structured chapters and thorough exploration
[[Personal Blogs|Blog Posts]]: Static and defining something at a certain time
[[Continuous Notes|(Continuous) Note]]: Notes that continuously get updated in a system such as a [[Second Brain]] or [[Digital Garden]].
- Highlighting and notes on external reads: Thinking that influences and adds to continuous notes. Synced with tools such as [[ReadWise]] to your second brain.
[[Academic and Formal Writings]]: Structured content following specific formats for particular purposes
- [[Essay]]: Extended, structured arguments or explorations of specific topics
- [[Abstract]]: Concise summaries of longer works
- [[Manifesto]]: Public declarations of intentions, motives, or views
- [[Thesis]]: Extended academic arguments supporting a specific point of view
- [[White Paper]]: Authoritative reports on complex issues
- [[Case Study]]: Detailed analyses of specific instances or examples
[[Exploratory Content]]: Content focused on thorough investigation and discovery
- [[Deep Dive]]: Comprehensive, detailed explorations of specific topics, often less formal than essays and potentially incorporating multimedia elements
- [[Research Journal]]: Ongoing documentation of exploration into a particular topic
- [[Explainer]]: Content that breaks down complex topics into understandable components
- [[Tutorial]]: Step-by-step instructional content focused on practical application
[[Reference Materials]]: Content designed primarily for consultation rather than sequential reading
- [[Wiki]]: Collaborative, interlinked knowledge repositories
- [[Documentation]]: Technical explanations of how systems or products work
- [[Glossary]]: Collections of definitions and terminology. See [[Data Engineering Glossaries]].
- [[FAQ]]: Organized answers to common questions
Are there more [[Types of Writings]]?

Type of Notes

See [[Type of Notes]] with Fleeting Notes, Literature Notes and Permanent Notes.

Ratatui

Thu, 31 Jul 2025 09:18:25 +0000

A [[Rust]] crate for cooking up Terminal User Interfaces ([[TUIs]]).

What is Ratatui?

Ratatui is a Rust library for cooking up delicious TUIs (terminal user interfaces). It is a lightweight library that provides a set of widgets and utilities to build simple or complex rust TUIs.

Data Engineering Conferences and Events

Mon, 07 Jul 2025 17:09:46 +0000

Data events around the world.

US

Dedicated data conferences

[[Data & AI Summit]] (Databricks)
[[Snowflake Summit]]
[[Data Council (Conference)]]
[[dbt Coalesce Conference]]

Others:

Infrastructure as Code (IaC)

Mon, 23 Jun 2025 14:06:32 +0000

Infrastructure as Code (IaC) is the practice of managing and provisioning IT infrastructure using code instead of manual processes.

It treats infrastructure like software, defining it in machine-readable definition files. This allows for automation, consistency, and version control of infrastructure deployments, similar to how software is developed.

Data Engineering Lifecycle

Fri, 20 Jun 2025 07:02:13 +0000

In today’s landscape, a data engineer is pivotal in overseeing the entire data engineering process. This involves gathering data from diverse sources and ensuring its availability for downstream applications. A deep understanding of the various stages in the data engineering lifecycle is essential. Additionally, a data engineer must possess the skill to evaluate data tools effectively, considering various aspects such as cost, speed, flexibility, scalability, user-friendliness, reusability, and interoperability.

Essential Data Engineering Toolkit

Thu, 19 Jun 2025 19:00:19 +0000

From Linux Basics to DuckDB: With the goal that beginners can improve their end-to-end data engineering skills. Tools would start from Linux basic commands, Dockerfiles and builds, ENV variables, Python tools, and DuckDB for everything SQL. Everything a data engineer needs to know today.

Embeddings

Thu, 19 Jun 2025 14:09:18 +0000

Embeddings are numerical representations of data (text, images, etc.) in a high-dimensional vector space where similar items are positioned closer together. For example, the words “happy” and “joyful” would have similar vector representations because they’re semantically related.

Retrieval-Augmented Generation (RAG)

Thu, 19 Jun 2025 14:04:06 +0000

Retrieval-augmented generation is a technique that can provide more accurate results to queries than a generative large language model on its own because RAG uses knowledge external to data already contained in the [[Large Language Models (LLMs)]].

Lakebase (Managed Postgres)

Wed, 18 Jun 2025 12:52:50 +0000

Announced at the [[Databricks Data & AI Summit 2025]] on the [[2025-06-11]].

Essentially an OLTP ([[Postgres]]) for the [[Data Lakehouse|Lakehouse]], powered by [Neon]. Although it might only be the compute, a fully managed Postgres, nothing based on Object storage.

Kestra

Tue, 17 Jun 2025 15:27:32 +0000

Kestra is a universal open-source orchestrator that makes both scheduled and event-driven workflows easy. By bringing [[Infrastructure as Code]] best practices to data, process, and microservice orchestration, you can build reliable workflows and manage them with confidence.

Rust

Wed, 11 Jun 2025 19:16:10 +0000

Rust is a versatile, general-purpose programming language known for its emphasis on performance, type safety, and concurrency. A key feature of Rust is its enforcement of memory safety, ensuring all references are valid without the need for a garbage collector or reference counting, common in other memory-safe languages.

Transaction Log in Open Table Formats

Thu, 05 Jun 2025 15:19:50 +0000

Transaction logs are the fundamental backbone of modern [[Data Lake Table Format|Open Table Formats]], enabling ACID transactions, metadata management, time travel capabilities, and concurrent operations on [[Data Lake|Data Lakes]]. This note explores how transaction logs are implemented across three major open table formats: Delta Lake, Apache Hudi, and Apache Iceberg.

Data Catalog

Mon, 02 Jun 2025 15:53:10 +0000

In our data-driven world, the volume of data is expanding at an unprecedented rate. Remarkably, 90% of the world’s data has been generated in just the past two years. Managing and organizing this rapidly growing data can be daunting. This is where a Data Catalog becomes essential.

Finding Flow (Focus)

Thu, 29 May 2025 06:41:20 +0000

I wrote a full article on Finding Flow: Escaping Digital Distractions Through Deep Work and Slow Living.

Very related is also [[Healthy Food]]. We all know that we should eat healthy, but mostly it’s more convenient to eat unhealthy, fast food.

Openness (Open Data Architecture)

Wed, 28 May 2025 08:17:42 +0000

Openness is when you bet on sustainability and open, trustworthy data architectures through open-source and standardized formats.

What is Openness?

Openness is an environment where trust and visibility flourish, mainly through [[Open Source]]. The word open, which is part of the word, is vital. While open doesn’t mean all the code is open source on GitHub, primary pillars are based on open source.

Data Contract CLI: CLI to manage your datacontract.yaml files

Fri, 23 May 2025 09:07:26 +0000

The datacontract CLI is an open source command-line tool for working with Data Contracts. It uses data contract YAML files to lint the data contract, connect to data sources and execute schema and quality tests, detect breaking changes, and export to different formats. The tool is written in Python. It can be used as a standalone CLI tool, in a CI/CD pipeline, or directly as a Python library.

Ballista (Arrow)

Mon, 19 May 2025 20:12:02 +0000

Ballista: Distributed SQL Query Engine, built on Apache Arrow. Competes with [[Apache Spark]]. It’s basically a scheduler for [[DataFusion]] on top of Apache Arrow.

GitOps

Wed, 14 May 2025 13:14:13 +0000

This is a specification of [[DevOps]] where tools like [[Flux]], [[ArgoCD]], [[Terraform]], or [[GitHub Actions]] are used for automatic deployment.

Github Codespaces

Wed, 14 May 2025 09:53:41 +0000

Blazing fast cloud developer environments (Cloud [[IDE]]). [[VSCode|Visual Studio Code]] backed by high-performance VMs that start in seconds.

Similar to [[replit]] and the next level to [[Jupyter Notebook]]s.

Devpod

Wed, 14 May 2025 09:52:06 +0000

[[Github Codespaces]] but open-source, client-only and unopinionated: Works with any [[IDE]] and lets you use any cloud, kubernetes or just localhost docker. Open Source Dev-Environments-As-Code.

Airflow DAG Factory Pattern

Tue, 13 May 2025 17:08:44 +0000

Dynamically generate [[Apache Airflow]] [[DAG]]s from [[YAML]] configuration files. A [[declarative]] way of using Airflow.

Benefits

Construct DAGs without knowing Python
Construct DAGs without learning Airflow primitives
Avoid duplicative code
done with YAML

Quickstart Example

Source

When in doubt, Tidy up

Mon, 12 May 2025 08:33:51 +0000

[[Austin Kleon]] says when he is stuck, he cleans up. The best cleaning is exploring, as he finds unfinished stuff in his “organized mess”.

Brain Dump

Mon, 12 May 2025 08:28:14 +0000

Pouring everything on paper that is currently in your mind. Freeing thinking power for other tasks. Usually helpful to avoid [[multitasking]].

Clarity

Mon, 12 May 2025 08:27:21 +0000

Clarity comes from having time to think, read, and create. The outcome is finding YOUR truth and removing resistance more deeply.

Webmention.io

Mon, 05 May 2025 15:07:31 +0000

Webmention.io is a hosted service created to easily receive web mentions on any web page.

How: Use it on your site

Once you have signed up, add the following tags to your HTML, replacing “username” with the username you got when you signed up:

Linter for TitleCasing Headers in Obsidian

Wed, 30 Apr 2025 19:48:44 +0000

For my articles I write, I always use Title Capitalization Tool to have consistent titles for my headers:

![[img_Linter for TitleCasing Headers_1746038418734.webp]]

Time-Travel

Wed, 30 Apr 2025 19:21:39 +0000

Time travel is a powerful feature in [[Data Lake Table Format]] that enables versioning of big data stored in your [[data lake]]. This capability allows you to access any historical version of your data, providing robust data management through:

INFORMATION_SCHEMA

Wed, 30 Apr 2025 13:08:52 +0000

INFORMATION_SCHEMA provides access to database metadata, information about the MySQL server such as the name of a database or table, the data type of a column, or access privileges. Other terms that are sometimes used for this information are data dictionary and system catalog.

Different Levels of Data Modeling

Fri, 25 Apr 2025 12:12:31 +0000

From 20 years of [[Data Modeling]] I learned that that before we choose between Inmon and Kimball, but today, there are so many layers, levels, and approaches.

Data Manipulation Language (DML)

Wed, 23 Apr 2025 09:52:49 +0000

Data Manipulation Language (DML) is a subset of [[SQL]] used to manage data within database objects created by DDL statements.

Data Definition Language (DDL)

Wed, 23 Apr 2025 09:39:36 +0000

A Data Definition Language (DDL) is a subset of [[SQL]] used to define and modify the structure of database objects such as tables, views, indexes, and schemas.

Snowflake Transactions Handling

Tue, 22 Apr 2025 08:32:56 +0000

By default, Snowflake operates in autocommit mode, where each statement is automatically committed after execution.

When your client changed it to manual, they switched to multi-statement transactions, where you need to explicitly commit your DML operations.

Schema Evolution

Tue, 22 Apr 2025 06:32:41 +0000

In the realm of [[data lake table format]]s, Automatic Schema Evolution emerges as a pivotal feature. It addresses the persistent challenge of modifying formats in the data engineering landscape. Essentially, Schema Evolution enables the seamless addition of new columns or the expansion of data types, all without disrupting existing structures.

What language does an Orchestrator speak?

Thu, 17 Apr 2025 06:04:46 +0000

Also, what language is best suited for [[Data Orchestrators]]? As all of the above data orchestrators speak Python, SQL is rising again. The post about SQL vs. Python for Data Analysis shows that both should be used in conjunction and whatever fits better. For example, using dbt for SQL transformation and a schedule and orchestrator to integrate them with [[Prefect]] or [[Dagster]] gives you the best of both worlds. I want to say believe that there is almost no data engineering project that does not use Python at some point.

Migrating from OneNote to Hugo

Wed, 16 Apr 2025 07:51:38 +0000

How the migration worked

I migrated as quick as I did because of how convenient it was to export the entire content I had written over time. I used wordpress-to-hugo-exporter, which is one of the provided plugins on GoHugo. After exporting, I had all my post as single markdown files and all images linked to them.

Use notebooks to open up the data silos

Fri, 11 Apr 2025 07:34:46 +0000

Data Notebooks popularized by Jupyter Notebooks are the centralized [[IDE]] inside a browser for doing collaborative work.

Tools

Some notebooks that exist today in different evolutions:

Writing is Hard

Thu, 10 Apr 2025 10:07:10 +0000

Everyone who can write has a competitive advantage because not everyone can do it. [[Tim Ferriss#Ryan Holidays]] said that when he felt competitive in angel investing or podcasting, he was thinking of going back to [[writing]].

Data Engineer vs. Software Engineer

Fri, 04 Apr 2025 13:59:03 +0000

There is no general answer as every role differs slightly depending on country and company.

[[SDLC (Software Development Lifecycle)]].

However, they are uniquely different, although it helps a ton to have SW engineering skills as data engineering adopts more and more techniques. But the fundamental work is still modeling, cleaning, transforming data, and creating business value.

Stoic

Thu, 03 Apr 2025 20:22:45 +0000

it’s a [[Philosophy]] designed to make us more resilient, happier, more virtuous and wiser–and as a result, better people, better parents and better professionals. Sounds a bit utopic? Maybe it is, but it helped me early in my personal life and career. I first heard of it by, again, Tim Ferriss and all his guests reading it. Famous leaders have been practising and shaping it, such as [[Marcus Aurelius]], the emperor of Rome, [[Theodore Roosevelt]], [[Epictetus]], [[Seneca]], to name a few.

Real-Time Analytical Database

Fri, 28 Mar 2025 09:32:11 +0000

Conversely, real-time analytical databases like ClickHouse, Pinot, Druid, and DuckDB achieve higher performance by co-locating compute and storage[^1]. These systems keep frequently accessed data and aggregations in memory, enabling extremely fast query performance, similar to what the initial OLAP solutions such as SSAS and others did best initially.

Vim Anywhere

Wed, 26 Mar 2025 20:24:40 +0000

I’m also super interested in this topic. I am a heavy [[Vim Language (and Motions)|Vim Motions]] user and wrote about my vim-verse, and I use [[Markdown]] whenever I can use [[Neovim]] and [[Obsidian]].

Map of Content (MOC)

Wed, 26 Mar 2025 19:40:09 +0000

These are meant to cluster the Zettelkasten notes into more prominent clusters, connecting the more significant ideas rather than individual notes. Instead of folders, we can use these as a starting point to explore or to read about topics that interest us.

MacOS key repeat for keyboard beyond normal settings

Mon, 17 Mar 2025 21:26:40 +0000

This is the fastest you can set when going into settings:
![[Pasted image 20230919091547.png]]

But there is an even faster way; you can change the preference keys by running the following in Terminal:

Why Vim Is More Than Just An Editor – A Comprehensive Review

Mon, 17 Mar 2025 20:40:45 +0000

Throughout my time as a developer, I’ve used [[VSCode]], [[Sublime]], [[Notepad++]], TextMate, and others. But shortcuts like cmd(+shift)+end and jumping with option+arrow-keys from word to word needed to be faster at some point.

Type of Notes

Tue, 11 Mar 2025 16:16:32 +0000

Besides the content format, there is also the type of notes introduced by the [[Zettelkasten]] method by [[Niklas Luhmann]]:

[[Fleeting Notes]]
[[Literature Notes]]
[[Permanent Notes]]

Origin:

Continuous Notes

Tue, 11 Mar 2025 16:06:56 +0000

This is the essence of a [[Second Brain]] and the [[Future of Blogging]] of blogging. Instead of old blogs that hold a wealth of information but need to be updated over time, continuous notes are updated frequently and win over time instead of losing value.

Default Path

Thu, 06 Mar 2025 20:26:28 +0000

The default path is a “script” of decisions and accomplishments needed to be seen as a successful adult. For example good grades -> good job -> own a home -> have a family

Data Product

Wed, 26 Feb 2025 13:44:16 +0000

Synonym for [[Data Assets]]. Started in [[Data Mesh]].

DJ Patil, former United States Chief Data Scientist, defined a data product as:

2025-02 SELECT Insights - Happy New Year

Sun, 23 Feb 2025 09:08:16 +0000

Hi there

I hope you’re doing well. I’m here with a late Happy New Year and the first newsletter of 2025 🙄. I hope you started well and have already achieved some of your goals. I didn’t set myself goals this year. Instead, I am doubling down on my Principles. I have firmly believed in them ever since I began using them.

anxiety

Wed, 19 Feb 2025 13:10:21 +0000

Don’t forget, we are all human.

It’s normal to have anxiety, not to know what comes next, to have negative emotions, to feel overwhelmed. When that happens, try to slow down. Say no to all things. Go out in [[Nature]], be present with your family, and recharge!

Slowly Changing Dimension - Type 2 (SCD2)

Tue, 18 Feb 2025 08:28:58 +0000

Slowly Changing Dimension Type 2 (SCD2) is a fundamental [[Data Warehouse|Data Warehousing]] technique that preserves the historical changes in dimensional data over time.

Showcase: This is my test

Tue, 18 Feb 2025 06:42:34 +0000

Let’s write a test note that I’m going to upload quickly.

This test was made to showcase [[Record Shell Prompts]] with asciinema. I made this video How I upload to my Public Second Brain.

Slowly Changing Dimension

Mon, 17 Feb 2025 12:20:02 +0000

Start of Slowly Changing Dimension.

A great demo How to Track History in a Data Warehouse (Slowly Changing Dimensions) - YouTube.

Schema Registry

Sun, 16 Feb 2025 13:20:24 +0000

Schema Registry provides a centralized repository for managing and validating schemas for topic message data, and for serialization and deserialization of the data over the network.

Schema Modification Operators (SMOs)

Sun, 16 Feb 2025 13:11:50 +0000

Schema Modification Operators (SMOs) are formal, operations that specify how to transform one database schema into another. Think of them as a standardized vocabulary or set of commands for making schema changes.

Schema Drift

Sun, 16 Feb 2025 13:08:02 +0000

A related concept discussed in [[RW Concept Drift and Model Decay in Machine Learning]]:

Its predictive ability decays over time. Broadly, there are two ways a model can decay. Due to data drift or due to concept drift. In case of data drift, data evolves with time potentially introducing previously unseen variety of data and new categories required thereof.

Recap (DB-Schema)

Sun, 16 Feb 2025 13:05:24 +0000

Recap is a dead simple data catalog for engineers by [[Chris Riccomini]]. GitHub.

Recap makes Schemas made easy. Work with your web service, database, and streaming schemas in a single format.

Protobuf

Sun, 16 Feb 2025 13:01:47 +0000

Protocol Buffers (a.k.a., protobuf) are Google’s language-neutral, platform-neutral, extensible mechanism for serializing structured data. You can find protobuf’s documentation on the Google Developers site.

Foreign Key

Sun, 16 Feb 2025 12:27:34 +0000

Foreign keys are pivotal in establishing a link between data across different tables. Essentially, a foreign key in one table points to a column in another table, typically the primary key, containing unique values. This setup facilitates a robust cross-referencing mechanism between the tables.

DBML

Sun, 16 Feb 2025 10:31:06 +0000

DBML (Database Markup Language) is an open-source [[DSL (Domain Specific Language)|DSL]] language designed to define and document database schemas and structures. It is designed to be simple, consistent, and highly readable.

Data Locality

Sun, 16 Feb 2025 10:26:28 +0000

Data locality refers to the principle of keeping data as close as possible to where it’s being processed to minimize data movement and access latency. This concept is crucial for optimizing performance in distributed systems and big data processing.

CAP Theorem

Sun, 16 Feb 2025 10:13:08 +0000

![[CAP theorem-20240910112512674.webp]]
source

From CAP theorem - Wikipedia:

In database theory, the CAP theorem, also named Brewer’s Theorem after computer scientist [[Eric Brewer]], states that any distributed data store can provide only two of the following three guarantees

Apache Avro

Sun, 16 Feb 2025 10:09:13 +0000

Avro is an open-source data serialization system that helps with data exchange between systems, programming languages, and processing frameworks. Avro helps define a binary format for your data, as well as map it to the programming language of your choice.

ACID Transactions

Sun, 16 Feb 2025 10:07:13 +0000

An ACID Transaction secures that either all changes are successfully committed or rollbacked. It makes sure you never end in an inconsistent state. There is different concurrency control that, for example, guarantees consistency between reads and writes. Each data lake table format has other implementations and features here. Read more on the respective [[Data Lake Table Format|Table Format]].

Factory Pattern

Sun, 16 Feb 2025 10:00:51 +0000

Factory Pattern Definition:

In software design, the “factory pattern” allows you to create a class, with its subclasses deciding which class to instantiate. Factory patterns allow the creation of objects without specifying the exact class of object that will be created, promoting loose coupling and scalability in code. This pattern is often implemented using a factory method or a factory class to produce instances of objects based on input criteria or conditions. The Factory Pattern is known as a “creational design pattern” (explained later in this article). Source

Principles

Sun, 16 Feb 2025 08:29:34 +0000

Principles are the key to living life. Find the right principles for you, and align your life accordingly, make small adjustments. And you will be in a better spot in one year, a huge better spot in two to five years. These kinds of small changes [[compounding]].

Never-Ending Now (Today's News)

Sat, 15 Feb 2025 17:13:09 +0000

We live in a never-ending now. It’s true; if you are on Instagram, Twitter, LinkedIn, Snapchat, god forbid, TikTok, you consume content that is produced today.

Daily Notes

Fri, 07 Feb 2025 23:29:52 +0000

This is a practice that usually includes an extension to open up today’s note with the annotation [[2024-08-05]]. You then have a template and start filling in some questions, maybe what you want to achieve for the day, and maybe notes or random thoughts during the day.

Traditional OLAP Cubes

Fri, 07 Feb 2025 15:20:56 +0000

Opposite to [[Modern OLAP Systems]], traditional ones were closed-source and used longer ago. They won’t use modern data engineering best practices. They are hard to integrate with git, version, and other benefits that more modern have.

Data Modeling Techniques

Thu, 06 Feb 2025 14:17:27 +0000

Explore various Data Modeling Techniques to enhance your understanding and application of data structures:

[[Relational Model|Relational Modeling]]: The origin of all modeling techniques.
[[Dimensional Modeling]]: A technique for organizing data to support fast retrieval of aggregated data.
[[Data Vault]] Modeling: A detail-oriented, historical tracking, and uniquely linked set of normalized tables that support one or more functional areas of business.
[[Anchor Modeling]]: An approach that provides agility in managing changes and historical data over time.
[[Bitemporal Modeling]]: Allows tracking both the historical and the system record of data.
[[Entity-Centric Data Modeling (ECM)]]: Focusing on the central entity, it offers two sub-techniques:
- [[Focal Modeling]]: Centered around a focal point, offering clarity in complex data structures.
- [[Activity Schema]] (Event-centric modeling): Designed to model and record activities, emphasizing the dynamic aspects of the business process.
[[Unified Star Schema]] or [[Unified Star Schema|Puppini Bridge]]

See Enterprise data warehouse (inmon and kimball), data vault and big table:
![[Enterprise Data Warehouse Modeling Inmon vs Star Schema Kimball vs data vault modeling and OBT.png]]
Also see [[Inmon vs Kimball#Comparative Insights]]

Serviced Cloud and Analytics

Wed, 05 Feb 2025 14:51:35 +0000

[[Looker]] ( Use-case where they replaced MS Analysis Cube with Looker)
[[Elastic cubes (Sisense) - Replacement of cubes]] ( ElastiCubes Technology)
[[panoply-io]]

The last option is to buy a Service Cloud Storage or Analytics vendor like [[Looker]], [[Sisense]], or [[panoply-io|Panoply]]. These are very easy to use and create implicit cubes for you, meaning you just join your data together in your semantic layer of the respective tool and all the rest is handled by the tool, including the reporting and dashboard tools. In this way you are more on the individual vendor, it might also be more expensive (prices are always very transparent and hard to get), but you are very fast up and running.

OwnStats

Wed, 05 Feb 2025 14:32:07 +0000

Ownstats is a self-hosted web analytics on AWS. Docs on Welcome to the OwnStats docs! | OwnStats Docs. Created by [[Tobias Müller]].

Website Analytics

Wed, 05 Feb 2025 14:21:14 +0000

There are a bunch of tools nowadays. Some come for free and some are open-source, some respect [[GDPR]] regulations, and some do not.

How data teams struggle to build an Analytics API

Sun, 02 Feb 2025 08:59:18 +0000

Cloud architecture is more complex than ever, especially with the latest explosion of tools and technology. Today, every data team wants data to be readily available to decision-makers in the company. Whether a Data Analyst, Product Manager, Data Scientist, Business or Data Analyst approaches them, it’s hard to provide a single interface to abstract all heterogeneous data stores away and let them query all the data. On top of that, new principles and architecture are picking up old ideas, for example, decentralised data products in [[Data Mesh]] and a centralised cloud data warehouse.

Learn in Public

Thu, 30 Jan 2025 13:02:30 +0000

The fastest way to learn

A term I first heard from [[Shawn Swyx Wang|Swyx]] on Learn In Public (might be originally from [[Tiago Forte]]) and captivated well by what I was doing with my blog over many years. All in the way of [[The more you share the more you get]].

The rise and history of Power BI with Amir Netz

Tue, 28 Jan 2025 16:18:53 +0000

In this episode of Kasper On BI we are talking about the history of Business Intelligence at Microsoft with one of the early founders and Technical Fellow at Microsoft Amir Netz.

Master Data Management (MDM)

Sun, 19 Jan 2025 19:02:37 +0000

Master Data Management (MDM) is a crucial methodology for centralizing [[Master Data]]. It serves as a pivotal bridge between business professionals, who are the primary custodians and experts of this data, and data specialists.

EcZachly Inc's Bootcamp

Fri, 17 Jan 2025 10:16:24 +0000

The Course Link to “Data Mastery” by [[Zach Wilson]].

About the DataEngineer.io Boot Camp

The DataEngineer.io Academy is your gateway to mastering data engineering! Whether you’re an analyst or an engineer, we’re here to help you level up your skills.

Dimensional Modeling

Fri, 17 Jan 2025 10:09:04 +0000

Dimensional Modeling (DM), integral to the [[Kimball DW and BI Lifecycle Methodology]], also known as Business Dimensional Lifecycle methodology, was developed by [[Ralph Kimball]]. It encompasses a suite of methods, techniques, and concepts essential for [[Data Warehouse]] design.

Jekyll

Thu, 16 Jan 2025 12:32:10 +0000

Another [[Static Site Generators (SSG)]].

Transform your plain text into static websites and blogs.

Built in built in [[Ruby]]. Ruby is great for web too (Claude). [[Jekyll]] was built by [[GitHub]] for this purpose

Writer Tools

Thu, 16 Jan 2025 08:32:27 +0000

Tools that help a writer.

Tools

[[Grammarly]] for checking Grammar
[[ChatGPT]] and other AI to brainstorm, fix errors, and ask for missing content. Never let it generate the text, always submit your text with each request.
[[Foster]]: Google Docs extensions wich real people giving you feedback.
Hemingway Editor highlights long parts and makes it enjoyable for the readers
![[hemingwayapp.png]]

Origin: [[How to Write Well]]

PDE (Personalized Development Environment)

Wed, 15 Jan 2025 12:46:03 +0000

It’s a term that [[TJ DeVries]] brought up with [[Neovim]] instead of only measuring speed. And also because [[IDE]] is not the right word as [[vim]] is not really an IDE. [[Neovim]] is a text editor but can do much more, even more than [[VSCode]] in certain use cases with [[LSP (Language Server Protocol)]] and [[Treesitter]].

Cron

Wed, 15 Jan 2025 10:50:10 +0000

The Cron command-line utility is a job scheduler found in Unix-like operating systems. Users who manage software environments often use cron to schedule jobs (cron jobs) to run automatically at specified times, dates, or intervals.

Backfill

Tue, 14 Jan 2025 20:23:18 +0000

“Backfilling is where you see the difference between a data engineer and a great data engineer.” — [[Christophe Blefari]] on LinkedIn

gRPC

Tue, 07 Jan 2025 09:10:56 +0000

gRPC is a cross-platform open source high performance remote procedure call framework. gRPC was initially created by [[Google]], which used a single general-purpose RPC infrastructure called Stubby to connect the large number of microservices running within and across its data centers from about 2001.

Folder Structure P.A.R.A.

Mon, 06 Jan 2025 15:28:37 +0000

There is now also a book: Amazon.com: The PARA Method: Simplify, Organize, and Master Your Digital Life eBook : Forte, Tiago: Kindle Store

Vim for Obsidian

Sun, 05 Jan 2025 12:25:45 +0000

Why using [[vim]]: [[Vim Language (and Motions)]]
how to active vim mode in [[Obsidian]]
why not use neovim only?
my workflow and shortcuts
My configs for Obsidian (hotkey, plugins, etc.)
1. .vimrc for Obsidian specific

My quick YouTube Video. Also, check my new long-form video [[Public Second Brain with Quartz#Long-form example of how I publish to Quart]].

Automatic Grammar/Spellchecking on Markdown files (DevOps)

Thu, 12 Dec 2024 12:42:54 +0000

Creating [[DevOps]] on Markdown, I thought could be a great idea for checking my blog/book, as it’s on GitHub already.

Activity Schema (Event-centric modeling)

Tue, 10 Dec 2024 13:59:54 +0000

A global standard for modeling and querying data

The Activity Schema™ data model is an open standard for data modeling and transformation. It enables simpler, faster, and more reliable data preparation. GitHub, activityschema.com

Quickly increment column of numbers in vim (count up)

Tue, 10 Dec 2024 09:01:48 +0000

See a video of it:
Quickly increment column of numbers with Vim

step by step

First, move the cursor to the top 0 (I use [] to signify cursor location).

Python and Functional programming

Thu, 05 Dec 2024 13:50:21 +0000

How to make your Python code more functional | by Ari Joury | Towards Data Science
Bye-bye Python. Hello Julia!. As Python’s lifetime grinds to a halt… | by Ari Joury | Towards Data Science
maps with python Python map() function - GeeksforGeeks
Why I Switched from Python to Clojure (2016) | Hacker News
[[(Functional) Data Engineering - Summary of blog posts]]

map()

map() function returns a map object(which is an iterator) of the results after applying the given function to each item of a given iterable (list, tuple etc.)

Graph Databases

Mon, 02 Dec 2024 09:01:58 +0000

A graph database is a database that uses graph structures for semantic queries with nodes, edges, and properties to represent and store data. A key concept of the system is the graph.

Feedback Loop

Tue, 26 Nov 2024 11:13:17 +0000

The feedback loop is the essence of why I [[Learn in Public]].

[[David Perell]] says that we consume only the [[Never-Ending Now]]; what is created today. Not a decade ago. This is one reason [[Why have your Website]] where you put out ideas, notes, blogs that stay over the long term. Creating [[Continuous Notes]].

AML

Thu, 21 Nov 2024 12:49:02 +0000

An easy [[DSL (Domain Specific Language)|DSL]] for database schemas. If you ever designed a database schema on a whiteboard, AML is made for you. It’s fast to learn and write, and can be translated to other dialects.

Azimutt

Thu, 21 Nov 2024 12:48:55 +0000

Azimutt · Database explorer and analyzer

Origin: [[AML]]

Personal Data Server (PDS)

Mon, 18 Nov 2024 10:34:44 +0000

Bluesky PDS (Personal Data Server) container image, compose file, and documentation. This is when you want to self-host your own [[Bluesky|Bsky]] server. Meaning the data is hosted on your server.

Headless CMS

Mon, 04 Nov 2024 16:54:12 +0000

Traditional [[Content Management System (CMS)]] architecture was once the standard for web development, but as business needs and customer expectations grow more complex, the more common technology used these days is known as Headless CMS, or Headless Content Management System.

Cube Store

Mon, 04 Nov 2024 15:27:03 +0000

The [[cache layer]] of [[Cube]]. Previously, they used [[Redis]]. Read more on [[RW Replacing Redis With Cube Store - Cube Blog]].

Semantic SQL: Bridging Data and Business Understanding

Fri, 01 Nov 2024 06:33:52 +0000

Semantic SQL represents the evolution of data querying and analysis, bridging the gap between raw data structures and business-friendly concepts. It emerged from the need to make complex data more accessible and understandable to business users without requiring deep technical knowledge of database structures or SQL syntax.

Kubernetes Hostings / Clusters (Serverless)

Tue, 29 Oct 2024 21:16:15 +0000

ovh cloud
Oracle
Hetzner
[[APPUiO]]
Okteto - 10 pods and 5 GB free
Fly.io
[[render.com]]?
[[Cloud Run]]?

Not Kubernetes, but to host simple apps [[Railway.app]] is amazing.

Data Lineage

Mon, 28 Oct 2024 15:35:36 +0000

Easily trace your data’s upstream (origin) & downstream (destination) dependencies.

Data lineage uncovers the life cycle of data. It aims to show the complete data flow from start to finish. Data lineage is the process of understanding, recording, and visualizing data as it flows from data sources to consumption. This includes all data transformations (what changed and why).

The Goal of Business Intelligence

Fri, 25 Oct 2024 14:25:23 +0000

But first, let’s talk a second about “what should BI do for us?”

In my terms, BI should produce a simple overview of your business, boost efficiency, and automate repetitive tasks across your organization. In more detail:

Reading

Mon, 21 Oct 2024 19:29:09 +0000

Reading is essential to [[Learning|learn]]. Without it, no learning.

Personal Journey

As I read all day at work, I’m a data engineer; I learned that I couldn’t read more books afterward. Even though my mind is still awake, my eyes are physically tired to read. They always fall close. Especially before bed. But I discovered that I can easily listen to a book for 20-45 more minutes.

Many-to-Many Relationships

Mon, 21 Oct 2024 17:55:08 +0000

In the realm of data warehousing, the concept of many-to-many relationships is pivotal, especially when it comes to understanding the nuances behind the purchase or sale of an item. Such relationships depict a scenario where a sale might be linked to multiple reasons, and conversely, a sales reason could be associated with various sales. This dual linkage is a core aspect of many-to-many relationships.

Bridge Tables

Mon, 21 Oct 2024 17:45:33 +0000

Bridge tables are crucial in dimensional modeling to represent [[many-to-many relationships]], where a fact table row is linked to multiple-dimension values and vice versa. This contrasts with the standard many-to-one relationship in dimensional models.

Cardinality

Sun, 20 Oct 2024 11:50:51 +0000

In the realm of [[Data Modeling|data models]], cardinality plays a pivotal role, referring to the uniqueness of data values within a specific column of a database table.

Data Modeling Frameworks

Sun, 20 Oct 2024 11:50:51 +0000

Besides [[Data Modeling Tools]], there are also helpful frameworks that help you model your data, asking the right questions.

ADAPT™ for OLAP

[[ADAPT]]] says that more than existing data modeling techniques like ER and dimensional modeling are required for OLAP database design. That’s why ADAPT is a modeling technique designed specifically for OLAP databases. It addresses the unique needs of OLAP data modeling. The basic building blocks of ADAPT are cubes and dimensions, which are the core objects of the OLAP multidimensional data model.

Pandas

Sun, 20 Oct 2024 11:50:51 +0000

Pandas is a software library crafted for the [[Python]] programming language, aimed at data manipulation and analysis. It particularly shines in providing data structures and operations for working with numerical tables and time series. As free software, it is distributed under the three-clause BSD license.

Functional Programming

Sun, 20 Oct 2024 11:02:32 +0000

Functional Programming is a style of building functions that threaten computation as a mathematical function that avoids changing state and mutable data. It is a [[declarative]] programming paradigm, which means programming expressively and declaratively as opposed to [[imperative|imperatively]].

Programming / Coding

Sun, 20 Oct 2024 11:01:21 +0000

Programming, to me, is a form of [[Art]].

I have often felt that programming is an art form whose real value can only be appreciated by another versed in the same arcane art; [[Ed Nather]], Source

Subtracting in Life

Sun, 20 Oct 2024 10:38:58 +0000

In life, you should subtract things, don’t add or say yes to many more. [[Hell Yeah or No]].

The least successful people I know run in conflicting directions, are drawn to distractions, say yes to almost everything, and are chained to emotional obstacles.

Taxonomy of note types

Tue, 15 Oct 2024 09:31:35 +0000

🚦 My Statuses
- 📬 Start any note, idea, something I read, or anything that comes to mind. Just some fleeting notes. This can also be deleted after a while
- 🗃/🌻 I worked on it a bit. I added many fleeting notes, brainstormed, elaborated a bit, and made some references.
- 🗃/📖 literature notes written. Not yet ready for the [[Permanent Notes]] / [[Evergreen Notes]], but [[Literature Notes]] are formulated in whole sentences and worked already a bit, or I’m just happy with the content.
- 🗃/🌳 Evergreen / Permanent Notes. These are my long-running notes that will end up in [[Zettelkasten]] core with my own words. Here I separated the literature notes into different ideas to follow the zettelkasten principle and link them together.
Type of Notes

Thinking

Mon, 14 Oct 2024 06:05:51 +0000

[[Writing]] is the thinking.
Walking is thinking.
Sleeping is thinking.
Exercising hard is thinking. ([[The Extended Mind (Annie Murphy Paul)#^0edd9a]])

The Data Engineering Cookbook

Fri, 11 Oct 2024 06:38:15 +0000

This book, created by [[Andreas Kretz]] openly, tries to tackle all parts of the [[Data Engineering]] lifecycle.

Software-Defined Asset

Thu, 10 Oct 2024 11:53:43 +0000

Software-defined Assets were first introduced by Dagster. They represent a novel, [[declarative]] approach to managing data and orchestrating its maintenance. This approach involves using code to define the data assets you want to create. These asset definitions are version-controlled through git and inspectable via tooling. This transparency allows anyone in your organization to understand the canonical set of data assets, reproduce them at any time, and lays the groundwork for asset-based orchestration.

Redis

Wed, 09 Oct 2024 12:07:26 +0000

Redis (Remote Dictionary Server) is an open-source, in-memory data structure store that is commonly used as a database, cache, and message broker. Its in-memory approach allows for extremely fast read and write operations, which is one of the primary reasons it has become the de facto standard for implementing a cache.

Key Performance Indicator (KPI)

Tue, 08 Oct 2024 18:15:32 +0000

A Key Performance Indicator (KPI) is a metric used to gauge the performance and success of an organization or specific activities it undertakes. These activities can range from projects and programs to products and various other initiatives.

Programming and Scripting

Wed, 02 Oct 2024 08:31:02 +0000

Programming and scripting are important in data engineering. Proficiency in at least one general-purpose programming language like (SQL), [[Python]], Java, or newer performance-critical application like Rust is needed.

Managed Data Stacks

Mon, 30 Sep 2024 12:59:01 +0000

Managed DevOps for your [[Modern Data Stack]], navigating around [[Kubernetes]] and Cluster management. I think of it as lego blocks, that can be stacked together.

Reverse ETL

Sun, 22 Sep 2024 13:55:08 +0000

In reverse ETL, the traditional roles of sources and destinations in the ETL process are reversed. Data is transferred from a centralized repository to third-party applications and platforms, typically operational systems designed for specific user groups such as sales, marketing, or customer support teams.

Normalization vs Denormalization

Sun, 22 Sep 2024 13:48:29 +0000

[[Normalization]] vs. [[Denormalization]] is explained best in the image below:

![[normalization-denormalization-illustration.png]]
This illustration effectively demonstrates how different modeling techniques operate. It’s a visual guide to understanding these concepts in practice. Source: GitHub - Data-Engineer-Camp/dbt-dimensional-modelling

Newsletter Link Collection

Sat, 14 Sep 2024 21:10:49 +0000

[[SELECT Insights - Practical Data Engineering Guide]]

Data Integration

Podcasts & Videos

Great Podcasts:
- Dagster and orchestration: Data Engineering Podcast: An Overview Of The Sate Of Data Orchestration In An Increasingly Complex Data Ecosystem
- With [[Maxime Beauchemin]] about orchestration: Data Engineering Podcast: Reduce Friction In Your Business Analytics Through Entity Centric Data Modeling
- dlt to ease data integration: Data Engineering Podcast: Eliminate The Overhead In Your Data Integration With The Open Source dlt Library
Video on [[DataFusion]] and [[Apache Arrow]]: D3L2: Discussing Rust, Ballista, Ray SQL, Data Fusion with Andy Grove

Tutorials & Best Practices

Misc

Origin: [[Newsletter]]

Neovim Setup

Tue, 10 Sep 2024 13:28:02 +0000

VS Code style

See [[Neovim set up like VS Code]].

Nvim Setup

Latest with Lazy.nvim even:
How I Setup LSP In Neovim For An Amazing Dev Experience - Full Guide - YouTube

Getting the Data – Scraping

Tue, 03 Sep 2024 08:53:18 +0000

[!warning] Disclaimer

Everything shown here is demonstrated for learning purposes only. Before you begin, make sure you don’t violate the copyright of any website and always be friendly when scraping.

Adrianos Decaf

Tue, 27 Aug 2024 04:54:19 +0000

![[Pasted image 20240130161657.png]]

Tried [[2024-05-03]].

Endlich ein geschmacklich überzeugender Kaffee, welcher nach dem «Swiss Water [[decaf]]»-Verfahren aufbereitet wurde. Was wir besonders toll finden an diesem Projekt mit der Rancho San Felipe sind die kurzen Wege zwischen den einzelnen Produktionsorten. Vom Anbauort Neria, wo die Rancho San Felipe zu Hause ist, sind es nur 20 km bis zu Descamex in Cordoba, welche die Bohnen entkoffeiniert. Der Weg zum Hafen in Veracruz ist auch nicht mehr weit und somit legt unser Decaf keine 150 km hinter sich, vom Anbau bis zur Verschiffung.

MapReduce

Fri, 09 Aug 2024 13:20:38 +0000

MapReduce is a programming paradigm model that enables massive scalability across hundreds or thousands of servers in a [[Hadoop]] cluster.

Bill Inmon

Fri, 09 Aug 2024 13:20:37 +0000

The Father of [[Data Warehouse]] often contrasted with [[Ralph Kimball]].

Introduction

Bill Inmon is a pivotal figure in the field of data warehousing. He is renowned for developing the concept of the data warehouse and is often mentioned in conjunction with Ralph Kimball, another key figure in data warehousing. While Inmon’s approach emphasizes a top-down design where the data warehouse is the centralized repository for all organizational data, Kimball advocates for a bottom-up approach using dimensional [[Data Mart|data marts]].

Matcha

Fri, 09 Aug 2024 13:20:35 +0000

It’s a Japanese green tea powder made from finely powdered dried tea leaves.

It has a slightly bitter, vegetal taste and a vibrant green color that results from the leaves’ high chlorophyll levels. It’s been the cornerstone of traditional Japanese tea ceremonies for centuries, but it recently became popular because of its health benefits.

Edgar F. Codd

Fri, 09 Aug 2024 13:20:33 +0000

Edgar Frank “Ted” Codd was an English computer scientist who, while working for IBM, invented the [[Relational Model]] for database management, the theoretical basis for [[Relational Databases]] and relational database management systems.

SqlDMB: SQL Database Modeler

Fri, 09 Aug 2024 08:07:30 +0000

Develop Databases Online without writing a single line of code.

SQLDBM offers an easy, convenient way to design your database for Snowflake, Redshift, MS SQL Server, PostgreSQL & MySQL absolutely anywhere on any browser, working away without the need for any extra database engine or database modeling tools or apps.

Toad

Fri, 09 Aug 2024 07:51:05 +0000

The #1 Oracle Database Tool. Kind of the Oracle DBA’s Tool of Choice back when I started my career. It had multiple handy features. Mainly used for [[PL-SQL|PL/SQL]]

How to Zoom on a Laptop like Trackpad or Mobile

Mon, 05 Aug 2024 13:03:51 +0000

! Mouse Pinch-To-Zoom - Chrome Web Store
- zoom with alt+scroll or rightclick+scroll

Origin: How to zoom in (not the entire website) with a mouse/keyboard? : r/edge

DBDiagram

Mon, 05 Aug 2024 09:34:47 +0000

Draw Entity-Relationship Diagrams, Painlessly. A free, simple tool to draw ER diagrams by just writing code. Designed for developers and data analysts.

PlantUML

Fri, 28 Jun 2024 06:34:27 +0000

PlantUML is an open-source tool allowing users to create diagrams from a plain text language. Besides various UML diagrams, PlantUML has support for various other software development related formats, as well as visualisation of JSON and YAML files. The language of PlantUML is an example of a domain-specific language

Evidence - SQL with Markdown

Mon, 10 Jun 2024 09:47:25 +0000

[[Business Intelligence]] for Modern Data Teams ([[Modern Data Stack]]). Beautiful reports synced to your data with just [[SQL]] and [[Markdown]]. Or they call themself now “Business Intelligence as Code”.

Counting Words written in Markdown

Tue, 04 Jun 2024 10:29:25 +0000

To document the progress of my book writing, I created a little script that outputs words written over time.

Markdown files in my Obsidian Vault:

Database Triggers

Wed, 29 May 2024 12:28:20 +0000

Triggers are used to [[Data Ingestion|ingest]] data into our data platform or [[Data Warehouse]].

Pros

Optimization: Triggers operate close to the data, leveraging database optimizations by vendors or the open-source community. This allows for writing efficient triggers tailored to the database system’s specific capabilities.
Data Integrity: A key advantage is maintaining data integrity. We won’t lose deletes or updates. Implementing this through triggers ensures consistency across all applications that manipulate the data instead of scattering the logic across various application codebases.

Cons

Complexity in Handling Row Counts: Triggers must always check the @@ROWCOUNT variable as the first implementation statement. This is crucial because triggers can fire even when no update or insert exists (e.g., where 1=2 conditions).
Nested Trigger Issues: When triggers cause other triggers to fire, managing row counts becomes complex. It’s often suggested to turn row counting off in such cases.
Delete Operations in Merge Statements: Special attention is needed when dealing with delete operations in merge statements. The @@ROWCOUNT in merge operations represents the total number of rows affected, not the count for individual insert, update, or delete actions within the trigger.
Error Handling: Implementing robust error handling within triggers, such as using TRY-CATCH blocks, is essential to prevent triggers from blocking inserts or causing unintended side effects. This requires careful construction of queries and validation of data.
Performance: If you mass insert/update to a table, it takes substantially longer to insert as each row insert will fire the trigger and its stored procedure.

In essence, while triggers offer advantages in terms of optimization and data integrity, they also come with their complexities, especially regarding row count management, nested trigger interactions, handling exceptional cases like merge statements, and ensuring robust error handling.

Architecture type (linux/amd64 and arm64)

Tue, 30 Apr 2024 17:12:00 +0000

Architecture types

arm64 –> arm64 (apple mac chips)
x86_64 –> amd64 (intel 64 bit CPU)
i386 –> 32 bit

MacOS arch type i386 Rosetta

Install Rosetta on MacOS:
softwareupdate --install-rosetta --agree-to-license

DevOps

Thu, 18 Apr 2024 11:11:36 +0000

DevOps represents a synergistic blend of software development (dev) and operations (ops). This methodology is characterized by its emphasis on integrating the efforts of both software development and operations teams. It fosters a culture that values collaboration and shared responsibility, pivotal for efficient workflow and productivity.

CDI - Customer Data Infrastructure

Tue, 16 Apr 2024 19:50:31 +0000

A Customer Data Infrastructure (CDI) solution equips organizations with prebuilt data connectors to efficiently gather data from various point solutions and route this data to requisite systems. Essentially, CDIs serve as a data pipeline, facilitating the movement of data from one system to another. Initially developed on the technical side—primarily to reduce the necessity for custom-built APIs—CDIs gradually integrate into business processes. While they offer limited data storage capabilities, it’s important to note that CDIs are not substitutes for a [[CDP (Customer Data Platform)]].

Fact Table

Tue, 16 Apr 2024 19:15:59 +0000

A fact represents the measurable, quantitative data of a business process.

It’s typically numerical and stored in a fact table, which references various [[Dimensions]] through foreign keys. Facts are the core metrics used for analysis and decision-making in a data warehouse environment.

XTable (formerly: OneTable)

Sun, 17 Mar 2024 07:52:15 +0000

Today [[2023-02-02]] [[Onehouse]] announced Onetable. Onehouse customers can now query their [[Apache Hudi]] tables as [[Delta Lake]] and [[Apache Iceberg]] unlocking native accelerations & advanced features inside Databricks and snowflake.

Temporal

Wed, 21 Feb 2024 08:11:46 +0000

Temporal simplifies and scales cloud application development, offering an open-source solution for writing and running reliable applications with less effort.

SQL Server Integration Services (SSIS)

Wed, 21 Feb 2024 08:06:08 +0000

Microsoft SQL Server Integration Services (SSIS) is a comprehensive platform designed for building high-performance data integration solutions, one of many [[ETL Tools]]. These solutions include extraction, transformation, and load (ETL) packages vital for data warehousing.

Object-Oriented Programming (OOP)

Wed, 21 Feb 2024 08:03:17 +0000

Object-Oriented Programming (OOP) is a programming paradigm that uses “objects” — data structures consisting of data fields and methods together with their interactions — to design applications and computer programs. It is built on the foundational concepts of classes and objects, encapsulation, inheritance, and polymorphism. OOP facilitates modular and reusable code, allowing developers to create more flexible, manageable, and intuitive software.

T-SQL

Wed, 21 Feb 2024 08:01:10 +0000

T-SQL, or Transact-SQL, is Microsoft’s extension to the SQL (Structured Query Language) standard, specifically tailored for use with [[Microsoft SQL Server]]. Designed to enhance SQL’s capabilities, T-SQL incorporates procedural programming, local variables, various support functions for string processing, date processing, mathematics, and more. It’s pivotal for managing Microsoft SQL Server databases and is embedded in tools like SQL Server Management Studio ([[SSMS]]) and Azure Data Studio.

PL/SQL

Wed, 21 Feb 2024 07:59:29 +0000

PL/SQL, short for Procedural Language for SQL, serves as Oracle Corporation’s procedural extension to SQL, enriching the [[Relational Databases|relational database]] experience. It’s integrated within [[Oracle Databases|Oracle Database]] starting from version 6, introducing stored procedures, functions, packages, and triggers by version 7. Additionally, it’s part of the Times Ten in-memory database from version 11.2.1 and has been adopted by IBM Db2 since version 9.7.

Unix Pipes

Wed, 21 Feb 2024 07:58:16 +0000

From the [[Unix Philosophy]].

The concept of communication between programs is pivotal, a challenge every novice programmer will encounter. On a Unix system, a simple yet effective method for data sharing is through pipes.

Data Assets

Thu, 15 Feb 2024 22:49:18 +0000

Data Assets are the result of the [[declarative]] pipeline in Dagster, as detailed in [[Software-Defined Asset]]. The power of this approach is discussed in [[Dagster#Why it’s so powerful]], highlighting how the declarative model is shaping the future of data.

Sönke Ahrens

Wed, 14 Feb 2024 08:47:15 +0000

Author or [[How to Take Smart Notes (Sönke Ahrens)]]
first to lumens [[Zettelkasten]] system and the benefit of not collecting notes, but using them interlinked

Origin: [[How to Take Smart Notes (Sönke Ahrens)]]
References: [[Zettelkasten]] [[Smart Note Taking]]

Materialized Views

Fri, 09 Feb 2024 15:11:00 +0000

A materialized view is a database object containing the results of a [[SQL]] query. The concept is also known as “Query rewrite”. It might be a local copy of remotely located data, a subset of a table’s rows and/or columns, a join result, or a summary created using an aggregate function.

Slice and Dice

Fri, 09 Feb 2024 15:10:58 +0000

Related to [[OLAP]] cubes, the concept of “slice and dice” refers to filtering [[Dimensions]] of data, much like manipulating a cube.

Data Mart

Fri, 09 Feb 2024 15:10:53 +0000

In today’s landscape, where big data and analytics reign supreme, data marts have emerged as a crucial tool for efficiently transforming vast amounts of information into actionable insights. Unlike [[Data Warehouse]]s, which are designed to handle massive datasets, data marts focus on making data easily accessible and readily available for [[Analytics]]. The rationale is simple: business professionals shouldn’t have to navigate complex queries to retrieve the data necessary for their reports. This is precisely where the strategic implementation of data marts by forward-thinking companies comes into play.

Postgres WAL Implementation

Fri, 09 Feb 2024 15:10:51 +0000

How it works see A Guide to Logical Replication and CDC in PostgreSQL with Airbyte - Neon.

A nice implementation with Singer Tap: pipelinewise-tap-postgres/tap_postgres/sync_strategies/logical_replication.py at d3180a193e30b3813ea71d3425b1fe8b2379896c · transferwise/pipelinewise-tap-postgres · GitHub.

Push-downs

Fri, 09 Feb 2024 15:10:23 +0000

Push-downs, also known as query pushdowns, are a technique where transformation logic is pushed to the source database. This approach minimizes the need to store data physically and reduces the data transfer load over the network.

Hadoop Distributed File System (HDFS)

Fri, 09 Feb 2024 15:09:49 +0000

HDFS is an open-source storage system that underpins [[Hadoop]]. It’s designed to store large volumes of data across a distributed cluster, providing high throughput access to this data.

Hadoop

Fri, 09 Feb 2024 15:09:44 +0000

Apache Hadoop is a collection of open-source software utilities that facilitates the use of a network of many computers to solve problems involving massive amounts of data and computation. It provides a software framework for distributed storage and processing of big data using the [[MapReduce]] programming model.

ELTP

Fri, 09 Feb 2024 15:09:38 +0000

Alternative to [[ETL]] and [[Reverse ETL]]. More marketing than reality?

In full, the ELTP model encompasses the following steps:

Extract data from raw data sources.
Load data into durable storage, such as a data warehouse or data lake.
Transform raw data into new datasets.
Publish data to downstream users and business applications.

EtLT

Fri, 09 Feb 2024 15:09:36 +0000

EtLT refers to Extract, small transform, Load, Transform, and can be thought of as an extension to the ELT approach to Data Integration, usually done in tools such as [[Airbyte]], etc.

Super Table

Fri, 09 Feb 2024 15:09:13 +0000

Super Table (ST) is an initiative at our company aimed at defining, building, and sharing high-quality data products. These products come with formal ownership and commitments.

Ultradian Rhythm

Fri, 09 Feb 2024 15:09:04 +0000

In chronobiology, an ultradian rhythm is a recurrent period or cycle repeated throughout a 24-hour day. E.g. the first 10 minutes of a 90-minute (Ultradian Rhythm) deep work block are always going to be hard. That’s okay.

Y42

Fri, 09 Feb 2024 15:08:52 +0000

Mission Control for Your Data Pipelines.

They re-organized around the naming, as of [[2023-12-21]], they call themself now an [[Data Orchestrators|Orchestration]] platform — probably similar to [[Ascend.io]]:

Halmak

Fri, 09 Feb 2024 15:07:37 +0000

This is a [[Keyboard Layout]] created with a Machine Learning algorithm to use the least amount of key travel. So 73% of you would write English if you are using the fingers laying on the home row already.

Imperative (Pipelines)

Fri, 09 Feb 2024 15:07:31 +0000

An imperative pipeline explicitly outlines how to execute each step in a sequential, procedural manner. In contrast, a [[declarative]] [[data pipelines]] approach doesn’t specify execution order. Instead, it allows each step or task to determine the most optimal time and method for execution.

The Three V's of Big Data: Volume, Velocity, and Variety

Fri, 09 Feb 2024 15:06:50 +0000

[[Big Data]] is a term that has gained significant traction in recent times, primarily characterized by the “Three V’s”: Volume, Velocity, and Variety. This concept was first introduced by Gartner Inc. analyst Doug Laney in 2001. Each ‘V’ represents a crucial aspect of Big Data:

Data Warehouse Blueprints- Business Intelligence in der Praxis (Dani Schnider, Claus Jordan, Peter Welker, Joachim Wehner)

Fri, 09 Feb 2024 15:06:44 +0000

Origin:
References: [[BI Blueprints - Trivadis]]

Markdown Drawing

Fri, 09 Feb 2024 15:05:34 +0000

[[_Excalidraw]]
[[Mermaid]]
- Draw Diagrams With Markdown - Typora Support
[[Obsidian Canvas]]

Not Markdown, but should be part of it as it has a similar [[declarative]] approach: twitter.com/szal_ai/status/1478729860252057606.

New Tabs in Obsidian with version 1.0

Fri, 09 Feb 2024 15:05:20 +0000

Obsidian 1.0 was released this week ([[2022-10-15]]).

The Sliding Panes (Andy Matuschak Mode) plugin is now a core of Obsidian and is called Stacked Tabs.

Obsidian Aliases and Links

Fri, 09 Feb 2024 15:05:16 +0000

Linking is a powerful tool within [[Obsidian]] and there are different ways of doing so. Lets have a look at some of them.

Test Note

Fri, 09 Feb 2024 15:05:16 +0000

what about sub headings

Origin:

Python `*args` and `**kwargs`

Fri, 09 Feb 2024 15:05:11 +0000

*args and **kwargs are special syntax in [[Python]] that allow you to pass a variable number of arguments to a function.

AutoGPT

Fri, 09 Feb 2024 14:58:33 +0000

An experimental open-source attempt to make GPT-4 (built by [[OpenAI]] and similar to [[ChatGPT]]) fully autonomous.

Auto-GPT is an experimental open-source application showcasing the capabilities of the GPT-4 language model. This program, driven by GPT-4, chains together LLM “thoughts”, to autonomously achieve whatever goal you set. As one of the first examples of GPT-4 running fully autonomously, Auto-GPT pushes the boundaries of what is possible with AI.

Dolly

Fri, 09 Feb 2024 14:58:33 +0000

Databricks’ Dolly, a large language model trained on the Databricks Machine Learning Platform.

Open [[Large Language Models (LLMs)]] model that runs locally.

Open-Assistant

Fri, 09 Feb 2024 14:58:32 +0000

OpenAssistant is a chat-based assistant that understands tasks, can interact with third-party systems, and retrieves information dynamically to do so.

Gatsby

Fri, 09 Feb 2024 14:56:43 +0000

A [[Static Site Generators (SSG)]], also call themself [[Headless CMS]].

There’s a reason why the world’s top-performing website teams use Gatsby. Whether your site has 100 pages or 100,000 pages—we’re obsessed with delivering dynamic web experiences with performance and security that scales.

Open Preview on MacOS such as `.md, .json,etc.`

Fri, 09 Feb 2024 14:56:42 +0000

Markdown preview is not supported (see How Can I Get “Quick Look” to Display Previews of Markdown (.md) Files? : r/MacOS), Glance helps with that.

xattr -rd com.apple.quarantine

Fri, 09 Feb 2024 14:56:42 +0000

It dismisses the prompt that the application is from a third party and you need to go into privacy setting to allow. Basically that I trust the source of the application:

Zola

Fri, 09 Feb 2024 14:56:42 +0000

A [[Static Site Generators (SSG)]] with [[Rust]]. Fast in a single binary with everything built-in.

GitHub

Origin:

Data & AI Summit 2023 US

Fri, 09 Feb 2024 14:56:34 +0000

[[2023-06-28]]

[!summary]

Advancing Spark - Data AI Summit 2023 Keynote Summary Day 2 - YouTube

Summary

Apache Spark 3.4

better monitoring, engine, everything better:
![[Pasted image 20230707092655.png]]

Anchor Modeling

Fri, 09 Feb 2024 14:56:25 +0000

Anchor Modeling is an agile data modeling technique that is specifically designed to handle evolving data structures. It is built around the concept of storing each attribute as a separate table, which allows for more flexibility when dealing with schema changes.

Bitemporal Modeling

Fri, 09 Feb 2024 14:56:25 +0000

A more niche, but still valid modeling technique is [[Bitemporal Modeling]].

Bitemporal modeling is a specialized data modeling technique designed to handle historical data along two distinct timelines. This approach enables organizations to access data from different vantage points in time and allows for the recreation of past reports as they actually appeared, as well as how they should have appeared, given any corrections made to the data after its creation. Bitemporal modeling is particularly useful in sectors like financial reporting, where maintaining accurate historical records is critical.

Focal Modeling

Fri, 09 Feb 2024 14:56:25 +0000

Another modeling technique, close to [[Data Vault]]?

![[Pasted image 20230619111804.png]]
From Data Modeling Meetup Munich: An Introduction to Focal with Patrik Lager - YouTube.

DAD Stack

Fri, 09 Feb 2024 14:55:07 +0000

Similar to [[Open Data Stack]] but focused on [[Dagster]], [[Airbyte]] and [[dbt]].

References

References on Hi all I just posted a new article about <https airbyte com dagster #random, If dbt is the “T” part of an “ELT”, what do you use for “EL”? : r/dataengineering,
Data Ingestion as Code - a showcase of managing managed services. Dagster Community Day - Dec 2022 - YouTube
or an implementation of the DAD stack I wrote (including Metabase) on [[The Open (aka Modern) Data Stack Distilled into Four Core Tools]] with the GitHub Repo [[Open Data Stack Project]] ( GitHub - airbytehq/open-data-stack: Open Data Stack Projects: Examples of End to End Data Engineering Projects).

Origin: If dbt is the “T” part of an “ELT”, what do you use for “EL”? : r/dataengineering

Plural.sh

Fri, 09 Feb 2024 14:46:31 +0000

Plural.sh is a unified application [[deployment]] platform that makes it easy to run [[Open Source]] software on [[Kubernetes]]. Instead of using [[Terraform]] to deploy to Kubernetes, which starts to make it complicated (with all Kubernetes) involved.

Stackable

Fri, 09 Feb 2024 14:46:30 +0000

The modular open source data platform: Popular data apps. K8s-native. Easy to deploy and run.

Open source data stack as of [[2022-09-09]]:
![[Pasted image 20220909093753.png]]

What has Rust in common with DuckDB?

Fri, 09 Feb 2024 14:45:53 +0000

I’m constantly thinking about what [[Rust]] and [[DuckDB]] have in common. I’m constantly interchanging them when thinking about one or the other, but why? 🤔

Arc Browser

Fri, 09 Feb 2024 14:45:48 +0000

Arc from The Browser Company

Feature

Zap: Cool new feature to remove elements

my Tweet

YouTube by [[Brandon Boswell]]

Before
![[Pasted image 20230526091904.png]]
After:
![[Pasted image 20230526091908.png]]

MS OneLake

Fri, 09 Feb 2024 14:45:40 +0000

Announced with [[Microsoft Fabric]] [[2023-05-23]] and it’s basically a wrapper around [[Delta Lake]] (for now). But they can also read from different Cloud providers such as AWS or Azure.

Location Independent Lifestyle?

Fri, 09 Feb 2024 14:44:06 +0000

Firstly, what is location-independent work or a nomadic lifestyle? I would describe it as not being tied to a particular location for any reason. And why would you want this? There are several answers besides the obvious you travel the world, learn from other cultures, and meet amazing people.

Pre-Game Routine

Fri, 09 Feb 2024 14:41:41 +0000

In order to get into the habit, it can help to develop a habit before the habit. [[James Clear]] calls that the pre-game routine.

Checking out PR (and branch) from existing PR (git worktree)

Fri, 09 Feb 2024 14:40:50 +0000

First, we need to configure to fetch PRs. We will add to .git/config which is located under root/config for [[git worktree]] repos:
![[Pasted image 20221222110245.png]]

How to not be terrible at writing

Fri, 09 Feb 2024 14:40:19 +0000

by [[Justin Gage]]

Writing should be fun and feel natural. Yes!:
Like code, writing should feel like a sort of flow – but it takes time to get there

Photo Sharing

Fri, 09 Feb 2024 14:40:15 +0000

Instead of [[Flickr]] which was used widely, an alternative is:

I’ve been using http://pixelfed.social - it federates with mastodon but looks like Instagram

Transcript of Tim Ferriss on How to Use Writing to Sharpen Your Thinking

Fri, 09 Feb 2024 14:39:56 +0000

I’m gonna talk a little bit about how to
use writing and why you should use
writing to improve your thinking so I
would credit much of the success I’ve
had to whatever extent I’ve had success
to learning how to write and to
practicing writing and the reason for
that is that without writing it’s very
hard to freeze your thinking on paper so
that you can sharpen it so that you can
see where for instance using words that
aren’t well-defined where you’re saying
things that don’t need to be said all of
this will help everything else I
remember in college for instance when I
took a class with John McPhee an
incredibly gifted writer and teacher all
of my grades and every other class went
up because every extraneous bit of
information that was hurting instead of
helping me was taken out so to sharpen
the salt a couple of different
approaches

Personalized API

Sun, 14 Jan 2024 10:36:34 +0000

On [[2023-01-25]], Hightouch announced their new offering: the Hightouch Personalization API.

This innovation piqued my interest, particularly due to its striking resemblance to the [[Analytics API]] which features GraphQL, API, and SQL interfaces.

Linking Your Thinking (LYT)

Sun, 14 Jan 2024 10:01:00 +0000

Build a personal system to manage your knowledge: The Linking Your Thinking Workshop is an online workshop that will teach, train, and support you in building a [[Personal Knowledge Management (PKM)]] system you can trust.

Zettel

Sun, 14 Jan 2024 09:55:57 +0000

It’s the small individual idea in the [[Zettelkasten]] system. [[Niklas Luhmann]] started this and he wrote each idea on an index card organized inside the Zettelkasten. These were thematically unlimited and extendable to infinity in any direction. Especially now with [[Obsidian]] and the [[Second Brain]], even more.

DAG: Directed Acyclic Graphs

Wed, 10 Jan 2024 12:58:36 +0000

DAGs, or Directed Acyclic Graphs, represent a conceptual or mathematical model of a data pipeline, embodying a series of activities in a specific arrangement.

Permanent Notes

Tue, 26 Dec 2023 13:21:00 +0000

Permanent notes are the third type of note in the [[Zettelkasten]] system, and make up the long-term knowledge that gives the slip-box its value. This step starts with looking through the first two kinds of notes that you’ve created: [[Fleeting Notes]] and [[Literature Notes]]. [[Sönke Ahrens]] recommends doing this about once a day before you completely forget what they contain.

How to Read Books

Mon, 18 Dec 2023 22:23:28 +0000

Tactical technique with skipping and choosing books on How to Read Faster by [[Mark Manson]]:

Quiet the inner monologue
Read with your finger
Stop reading books you hate
1. the point of reading is for you, don’t feel bad if you skip chapters or you don’t finish
Schedule time to read
Read more than one book at a time

Purely reading faster can be done with [[Speed Reading]] techniques.

When is a data engineer needed?

Wed, 06 Dec 2023 20:03:53 +0000

I believe, that not every company is in need of data engineers. His skills are mostly required if the company either:

DataOps

Wed, 06 Dec 2023 18:18:36 +0000

DataOps, in my opinion, is synonymous with the [[Data Engineering Lifecycle]]. It represents a culture that integrates and manages the lifecycle in a lean and agile manner, inspired by [[DevOps]] and Product thinking.

Use transactional processing

Wed, 06 Dec 2023 16:26:11 +0000

To support various features of a relational database in a data lake, you need transactional processing. Luckily [[Delta Lake]] is here to rescue you. Delta has many awesome features like [[ACID Transactions]], [[Time Travel]], keeping a [[Transaction Log (Delta Lake)]], SQL API to write native SQL as insert, update, delete, and even merge statements, open-format ([[Apache Parquet]]), unified batch and streaming source and sink (no [[Lambda Architecture]] needed anymore), [[Schema Evolution]], [[Optimistic Concurrency]]. The full list and more information on [[Delta Lake]]. Also, check out an excellent customer example with Apple about threat detection at scale.

Creativity vs Productivity

Sun, 03 Dec 2023 12:31:14 +0000

Creativity and [[Productivity]] are two of the same thing. You can’t be creative without being productive. It’s a pendulum that goes forth and back.

Asimba

Mon, 27 Nov 2023 19:30:08 +0000

![[asimba.png]]

Origin:

The more you share the more you get

Tue, 07 Nov 2023 11:12:34 +0000

The more you share the more you get. [[Leonard Nimoy]]

I’ve experienced the truth of this a long time ago during my apprenticeship, without being aware of it. You probably remember those days when you had to prepare for an exam, but due to time constraints, you couldn’t look up everything in the books. So my friend and I compiled summaries of the most important things. Not surprisingly, these summaries quickly became very popular in class, and more people asked us if we could share it with them. I remember that back then I was skeptical about this and was asking myself “Why should I give it to them? We put a lot of work into it and they just want to copy that? No way!” At the end of the day, they got it anyway either through my friend or some other way. Eventually, I just said to myself “Oooookaay, give it to everyone who asks,” and even without being asked, I would tell them that I have a summary for the next exam and sent it to whoever wanted it.

Senato (Entkoffeiniert)

Tue, 03 Oct 2023 08:10:44 +0000

Origin:

Gourmet Hörnan Special

Tue, 03 Oct 2023 08:10:42 +0000

Origin:

Zombo 04

Tue, 03 Oct 2023 08:10:38 +0000

![[image-1666357162060.jpg1917986521763527648.jpg]]

Origin:

ViCAFE Haus Mischung

Tue, 03 Oct 2023 08:10:35 +0000

![[image-1695536639858.jpg7746875548008160914.jpg]]

Origin:

Rwanda

Tue, 03 Oct 2023 08:10:33 +0000

![[image-1661925946779.jpg1946354368961512670.jpg]]

Origin:

Ristretto

Tue, 03 Oct 2023 08:10:30 +0000

![[Roesterei_kaffee_marzili.png]]

![[Untitled.png|Untitled.png]]

Origin:

Rio Verde

Tue, 03 Oct 2023 08:10:28 +0000

![[rio_verde.png]]

Origin:

Pintli Coffee

Tue, 03 Oct 2023 08:10:22 +0000

![[image-1685870018899.jpg645116421850910198.jpg]]

Origin:

Orang Utan

Tue, 03 Oct 2023 08:10:20 +0000

![[image-1619952328907.jpg1098685388604163437.jpg]]

Origin:

Nyota

Tue, 03 Oct 2023 08:10:18 +0000

![[image-1682487921109.jpg5363339763158360619.jpg]]

Origin:

Mövenpick

Tue, 03 Oct 2023 08:10:15 +0000

![[image-1682260050549.jpg7276034491896470969.jpg]]

Origin:

Mirador

Tue, 03 Oct 2023 08:10:12 +0000

![[image-1663427099083.jpg8811521245950969911.jpg]]

Origin:

Maytama

Tue, 03 Oct 2023 08:10:09 +0000

![[m.png]]

Origin:

Marsellesa

Tue, 03 Oct 2023 08:10:07 +0000

![[PXL_20210303_142535529.jpg]]

Origin:

Kent Cauwery

Tue, 03 Oct 2023 08:10:04 +0000

![[PXL_20210319_120817027.jpg]]

Origin:

Kangchen

Tue, 03 Oct 2023 08:10:00 +0000

![[image-1669212854481.jpg6488333567797760235.jpg]]

Origin:

Heirloom

Tue, 03 Oct 2023 08:09:57 +0000

![[PXL_20210303_142541746.jpg]]

Origin:

Galeras

Tue, 03 Oct 2023 08:09:53 +0000

![[image-1678862893628.jpg2014641335008618235.jpg]]

Origin:

GAB-Blend (Barista Edition)

Tue, 03 Oct 2023 08:09:50 +0000

![[gab.png]]

Origin:

Finca Dolores

Tue, 03 Oct 2023 08:09:47 +0000

![[image-1694943579516.jpg2524213861404112541.jpg]]

Origin:

Fansipan

Tue, 03 Oct 2023 08:09:45 +0000

![[image-1677153791108.jpg2565613938173499104.jpg]]

Origin:

Espresso la reine

Tue, 03 Oct 2023 08:09:42 +0000

![[image-1624694394235.jpg2461649275350291006.jpg]]

Origin:

Espresso House

Tue, 03 Oct 2023 08:09:39 +0000

![[image-1635329857234.jpg3931807076002760395.jpg]]

![[image-1635330022686.jpg4638618648085797489.jpg|image-1635330022686.jpg4638618648085797489.jpg]]

Origin:

Espresso dark roast

Tue, 03 Oct 2023 08:09:36 +0000

![[image-1627974876191.jpg4992429150935072620.jpg]]

Origin:

Espresso - Caffe in grani

Tue, 03 Oct 2023 08:09:33 +0000

![[image-1656324622708.jpg4851278771730615380.jpg]]

Origin:

Elgon

Tue, 03 Oct 2023 08:09:31 +0000

![[uganda.png]]

Origin:

Crema

Tue, 03 Oct 2023 08:09:07 +0000

![[exquisito-crema-bohnen-500g.jpg]]

Origin:

Cerro Chirropo

Tue, 03 Oct 2023 08:08:45 +0000

![[cherro.png]]

Origin:

Candelaria

Tue, 03 Oct 2023 08:08:42 +0000

![[image-1618650049455.jpg3681714176049737860.jpg]]

Origin:

Bandeira

Tue, 03 Oct 2023 08:08:35 +0000

![[bandeira.png]]

Origin:

Amaro Gayo

Tue, 03 Oct 2023 08:08:27 +0000

![[image-1669212960771.jpg2053001199999963551.jpg]]

Origin:

Dark Robusta

Tue, 03 Oct 2023 08:06:29 +0000

![[image-1691602249650.jpg458276504132409165.jpg]]

Origin:

Docusaurus

Tue, 03 Oct 2023 06:51:24 +0000

Build optimized documentation websites quickly, and focus on your content.

I believe it’s built on top of [[Gatsby]]. For E.g. similar can be achieved with other [[Static Site Generators (SSG)]] with specific themes. E.g. [[GoHugo#Docs]].

ngods (new generation open-source data stack)

Sun, 16 Jul 2023 09:20:48 +0000

As [[Modern Data Stack]] is an overused term, people searching for another one. One would be [[Open-Source Data Stack]] or this one is “New Generation open-source Data Stack”.

Restack.io

Mon, 19 Jun 2023 09:02:41 +0000

The simplest way to manage stacks of open source products on a production-ready [[Kubernetes]] cluster. Kind of a store where you chose your artifacts and restack will deploy it.

Productivity

Sun, 11 Jun 2023 12:50:01 +0000

Using your time intentionally. It’s efficient and minimizes waste.

For me has a lot to do with [[Personal Knowledge Management (PKM)]], but also goes deep into our life, that’s why my article about [[Personal Knowledge Management Workflow for a Deeper Life]] summarises it probably best up until this point.

Rich Text

Sat, 27 May 2023 08:24:20 +0000

The Rich Text Format is a proprietary document file format with published specifications developed by Microsoft Corporation from 1987 until 2008 for cross-platform document interchange with Microsoft products.

Speed Reading

Fri, 31 Mar 2023 08:13:50 +0000

Key points from How to Speed Read by [[Tim Ferriss]]:

Establish a baseline: Determine your words per minute (WPM) reading rate by counting the average number of words per line and lines per page in a standardized book. Read for one minute and calculate your WPM.
Use peripheral vision: Instead of reading from the far left to the far right, use your peripheral vision to focus on a narrower portion of the page. Draw lines on either side of the page, indent one word from each edge, and read between the lines.
Minimize fixation points: The eye doesn’t move smoothly across the page but jumps between fixation points. To minimize these points and reduce reading time, focus on fewer points per line.
Use a pacer: Use your finger or a pen to trace underneath the lines as you read, keeping your eye on the pacer to prevent back-skipping and maintain forward momentum.
Read slightly faster than comprehension allows: For five minutes, practice reading at a speed that causes a slight loss in comprehension. This helps reset your comfort set point for reading at full comprehension.
Retest your WPM: After implementing these techniques, retest your WPM and check for comprehension. Many people will see significant improvements in their reading speed.

These techniques are based on understanding the mechanics of the eye and optimizing optical perception for reading. They can help improve reading speed for various types of texts, from non-fiction to fiction and poetry.

Run Docker Images/Containers in interactice (`-it`) mode

Thu, 09 Mar 2023 17:20:07 +0000

Images

Either use:

`1`	`docker run -it <image> /bin/bash`

or, if this doesn’t work (when image has a defined ENTRYPOINT). For these cases use:

Data Engineering Approaches

Mon, 06 Mar 2023 10:02:04 +0000

Because I encountered these bottlenecks myself, and more frequently lately, I asked myself, how can we:

Make BI easier?
Transparent for everyone?
Effortless to change or add new data or transformations but still have some [[Data Governance]] and testing?
Fast in ad-hoc queries to explore and slice-and-dice your data?
Have more frequent data loads?
Simplify following the transformations for all data-savvy people and not exclusive for involved BI engineers.
Extend additional machine learning capabilities smoothly?

I’m aware that nowadays a lot is going on, especially around open-source tools and frameworks, data ops, and deployments with container-orchestration systems and the like.

Education is changing

Mon, 06 Mar 2023 09:55:14 +0000

Facebook, Airbnb, and other companies taking a step into so-called “Data Camps or Data University” to educate internal employees in respect of data to get more data savvy.

ETL is changing

Mon, 06 Mar 2023 09:54:21 +0000

Furthermore, the way we do ETL is changing as well, as [[Maxime Beauchemin]], data engineer at Airbnb quotes:

_Product know-how on platforms like Informatica, IBM Datastage, Cognos, AbInitio or Microsoft SSIS isn’t common amongst modern data engineers, and being replaced by more generic software engineering skills along with understanding of programmatic or configuration driven platforms like Airflow, Oozie, Azkabhan or Luigi. It’s also fairly common for engineers to develop and manage their own job orchestrator/scheduler.

ReadWise

Fri, 24 Feb 2023 18:55:47 +0000

Readwise can import from all kind of sources such as:

also with Instapaper
kindle app
Medium
and and and … etc.

it’s the app for all highlights in one place

Annual Work Reflection 2022

Tue, 03 Jan 2023 08:52:15 +0000

🎄 Soon is Christmas, and it’s always good to recap at the end of the year. The years always fly by like nothing, but a lot happened in the last year. My main goal is always to be true to myself and learn something new every day (if possible).

GitHub Copilot

Thu, 24 Nov 2022 09:52:42 +0000

The excellent tech can write not only code but also auto-complete comments. It uses GitHub repos for inspiration but can adapt to 1 bio or input params with GT3 (or similar?) and adjust names according to your comments or needs.

Literature Notes

Wed, 31 Aug 2022 06:34:17 +0000

The second is [[Literature Notes]]. [[Niklas Luhmann]] would write them down on index cards with the bibliographic details on the back. Each of them contains the main point he didn’t want to forget or that he thought he could use in his own writing. The bibliographic details are the 🌍 Wiki 📂 Files where my [[Reference Wiki Notes]] are stored.

Data engineer job description and skills

Sat, 27 Aug 2022 08:21:18 +0000

If English is the language of business, SQL is the language of data and Python is the language of engineering. While technology disappears often, SQL is still here. This means you need a reliable understanding of:

Minimalism

Sat, 27 Aug 2022 08:08:07 +0000

A lifestyle I admire. Not really crazy to have only 15 items or so, but that is before you buy something that you need to sell. Or that you have more money to buy quality things that you enjoy and use more instead of buying new things all the time.

Stop reading News

Sat, 27 Aug 2022 07:42:55 +0000

A long time ago, I stopped reading the news. Not reading the news doesn’t mean that I’m uninformed. But I stopped reading the newest, most attention-driven headline updated 100 times a day. As everyone probably knows, the news makes us depressed if we read it every day. They are rather focused on the negative or radical things instead of things that would make fun to read.

Fleeting Notes

Fri, 26 Aug 2022 19:44:28 +0000

Fleeting notes are quick, informal notes on any thought or idea that pops your mind. They don’t need to be highly organized and shouldn’t be. They are not meant to capture a picture in full detail but serve as reminders of what is in your head.

The Building a Second Brain methodology will teach you how to:

Fri, 19 Aug 2022 19:55:19 +0000

Consistently move your projects and goals to completion by organizing ahd accessing your knowledge in a results-oriented way
Transform your personal knowledge into income, taking advantage of a rapidly growing knowledge economy
Uncover unexpected patterns and connections between ideas
Reduce stress and “information overload” by expertly curating and managing your personal information stream
Develop valuable expertise, specialized knowledge, and the skills to deploy it in a new job, career, or business
Cultivate a collection of valuable knowledge and insights over time without having to follow rigid, time-consuming rules
Unlock the full value of the wealth of learning resources all around you, such as online courses, webinars, books, articles, forums, and podcasts

Most important take aways:

Coffee Beans Base

Mon, 01 Jan 0001 00:00:00 +0000

Coffee Machine: Solis Grind & Infuse Pro Typ 115 ([[Coffee Machines]])
Mühle Stufe innen: 6 (untere Angaben beim Kaffee ist die innere Stufe)

Second Brain: Crafted, Curated, Connected, Compounded on

Common Table Expression (CTE)

Will AI Replace Human Thinking? The Case for Writing and Coding Manually

Find Good Data Sets or Sources

Data Sets

Neomd

Semantic Layer

DuckDB

ssp.sh Newsletter

Promoting Your Work

Listmonk

CV

CV: Simon Späti

Data Engineer, Technical Author, and Educator

Introduction

Data Engineering Acquisitions (2022-2026)

Entity-Centric Data Modeling (ECM)

LTAP (Lake Transactional/Analytical Processing)

Zen Mode for Writing (Obsidian, Neovim)

Music that Keeps You Focused

Tokenmaxxing

Editing and Proofreading Your Writings

Classical Architecture of Data Warehouse: Staging -> Cleansing -> Core -> Marts

Death of the Follower: TikTokification

Vibe Coding: Think Slow, Cultivate Taste, and Build Fast

Writing with AI: What's the Best Approach?

The Differences between Writing and Coding

I'd rather Read the Prompt

How to Detect AI Writing

Data Modeling Languages: Query & Semantic Layer Evolution

Languages

Declarative vs. Imperative

DRY: Don't Repeat Yourself

Query Databases in Neovim (DBUI)

Static Site Generators (SSG)

Open-Source Data Engineering Projects (2022-2026)

Future of Blogging: More Interconnected

No Meetings: Why I Choose Async Over Short Calls

Digital Garden

My Note-Taking Path History and Evolution

Data Engineering Vault

Data Engineering Vault

My Today's Daily Graphs - Obsidian Graph

Graphs

Obsidian

Home Made Ramen: Step-by-Step Instructions

Data Model Engine

Ramen - Soy Sauce Eggs

Ingredients

Instructions

Data Engineering YouTube Channels

Plain Text Files

Kanagawa

Color Palette

Git Diff Report (HTML, txt)

TUIs

Tools

Development & APIs

AI

DevOps & Infrastructure

Database Management

Data & Streaming

System & Linux

File & Storage

Communication & Social Media

Productivity & Finance

Web & Information, RSS

Media & Downloads

Emails

SSH GUIs

Git and GitHub

Trains and Information

Lists

Example

Quartz: Publish Obsidian Vault

Open-Source Obsidian Publish Alternatives

Other Public Second Brains

Public Second Brain with Quartz

Convert Markdown to Rich Text (Google Docs)

Create a Book on Markdown

Set commands in `.vimrc`