But then again, you have to ask: was Python made for Data Engineering in the first place?

Rust may not replace Python outright, but it has consumed more and more of JavaScript tooling and there are increasingly many projects trying to do the same with Python/Data Engineering. Let’s explore why Rust has potential for data engineers, what it does well and why it has become the most loved programming language for 7 years running. 

What is Rust?

Former Mozilla employee Graydon Hoare initially created Rust as a personal project. The first stable release, Rust 1.0 was released on May 15, 2015. Rust is a multi-paradigm programming language that supports imperative procedural, concurrent actor, object-oriented and pure functional styles, supporting generic programming and metaprogramming statically and dynamically.

The goal of Rust is to be a good programming language for creating highly concurrent, safe, and performant systems.

What Is Unique about Rust?

Rust solves pain points from other programming languages with minimal downsides. With Rust being a compiled programming language, strong type and system checks are enforced during compile-time—meaning pre-runtime! Unlike Python’s interpreted way, most errors only surface during the coding phase. It can be frustrating to fight every single mistake before being able to test or run a quick script, but at the same time, a prominent feature like the compiler is much faster at finding bugs than me. Additionally, the Rust community puts a lot of effort into making the error messages super informative.

Why Rust for Data Engineers?

When you write any code, the goal is that the code doesn’t break during the weekend or at night when you sleep. Rust will show you errors and improvements while coding and fails as much as possible at compile-time, which is less costly than later in production at run-time.

The go-to language for data engineers is Python, which isn’t the most robust or safe language, as many engineers working with data will agree. Rust’s developer experience goes much further than just offering a language specification and a compiler; many aspects of creating and maintaining production-quality software are treated as first-class citizens in the Rust ecosystem

What Rust Does Well

Python is dynamically typed (with only recent support for type hints) and requires writing extensive tests to catch these costly errors. But that takes a lot of time, and you must foresee all potential errors to write a test for it.

Rust is the opposite; it forces you to define types (or does it implicitly with type inference) and enforces them. This does not obsolete testing of course, but for example, the rust compiler will analyze e.g. borrow checking, and does things to your code that other compilers don’t do—check out rust-analyzer for bringing them into your IDE of choice. This makes it very good for data engineers as we have many moving parts such as incoming data sets that we do not control. Defining expectations with data types and having vigorous checks at coding and compile time will prevent many errors.

Less relevant for data engineers, but super helpful: speed. Rust, as a compiled language, is super fast at run-time. To many, Rust is primarily an alternative to other systems programming languages, like C or C++. But you don’t need a systems use case to use a systems language, as both Vercel and Crowdstrike are noticing.

Another one is integrations. With data pipelines being the glue code in most cases, connecting otherwise foreign systems, Rust almost runs platform agnostic. Rust makes it easy to integrate and communicate with other languages through a so-called foreign function interface (FFI). The FFI provides a zero-cost abstraction where function calls between Rust and C have identical performance to C function calls. Rust can be called easily from C, Python, Ruby, and vice-versa. Find more on Rust Once, Run Everywhere.

A less technical but still important element is to love or have fun, enjoying your programming language. Rust is a more complex language to learn, but it was the most loved technology for seven years (2022, 2021, 2020, 2019, 2018, 2017, 2016) in a row on the Stack Overflow survey:

Besides the love, it’s also rising in awareness of different trends such as Google Trend, one from 2019 Ranking on GitHub, or the StackOverflow below:

Interesting Open-Source Rust Projects

The language is always only as good as its community. Let’s look at some of the existing open-source tools and frameworks built in and around Rust:

• DataFusion based on Apache Arrow: Apache Arrow DataFusion SQL Query Engine similar to Spark
• Polars: It’s a faster Pandas. Probably going to compete with DuckDB (?)
• Delta Lake Rust: A native Rust library for Delta Lake, with bindings into Python and Ruby
• Cube: Headless BI for Building Data Applications
  • Written mostly in Rust, Cube’s data processing and storage are based on the Arrow DataFusion query execution framework, which uses Apache Arrow as its in-memory format. Especially the core of Cube, the cache layer called Cube Store is 100% built-in Rust
• Vector.dev: A high-performance observability data pipeline for pulling system data (logs, metadata)
• ROAPI: Create full-fledged APIs for slowly moving datasets without writing a single line of code
• Meilisearch: Lightning Fast, Ultra Relevant, and Typo-Tolerant search engine
• Tantivy: A full-text search engine library
• PRQL: Pipelined Relational Query Language for transforming data
• Many more; please let me know of any

Less relevant to data engineering, but still cool:

• Deno: This is a fast Node.js version
• Tauri: Tauri is a framework for building tiny, blazingly fast binaries for all major desktop platforms
• Yew: A modern Rust framework for creating multi-threaded front-end web apps with WebAssembly.

Read more on a currated List of great Open-Source Rust Projects.

Rust vs. Python

The downside of Rust, the learning curve is much higher than other languages, such as Python. That’s why most Rust programs in data engineering will have a Python wrapper for integrating it into any Python data pipelines for a long time. It’s also a shift from an interpreted language such as Python to a more Functional Language (FP) style, which Rust certainly supports.

Other Recent Programming Languages

Newer programming languages follow the functional programming approach. New functional programming languages started, such as Scala with Akka, Elixir, or multi-paradigm programming languages such as Julia, Kotlin (a fastest-growing language since Google made it default for Android development), and Rust.

GoLang seems to be a good compiled programming language usedin DevOps.

Elixir has servers monitoring data pipelines and re-tries included in the language; no framework is needed. It makes an excellent fit for data engineering and would replace parts of the Data Orchestrators.

Rust as a Primary Language?

Let’s see an example of a modern data pipeline integrating with Airbyte, dbt, and some ML models in Python.

Each step can have errors and data mismatches. That’s why we have orchestrator frameworks such as Dagster, which force you to write functional code or the concept of Functional Data Engineering. There is also lots of adoption in Python with the type hint or writing more Python and Functional Programming style. Or to bring up an example of another language, JavaScript, the rise of TypeScript.

As we typically load data into a data frame and transform or add some business logic within our data pipelines. This could be done efficiently with Rust and Apache Arrow, and DataFusion, which is type-safe, and a good ecosystem. Time will tell.

Will Rust Be the Programming Language for Data Engineers?

Rust is a multi-use language and gets the job done for many problems of a data engineer. But the data engineering space is dominated by Python (and SQL) and will stay that way for the foreseeable future. There is no “until people fully move into Rust”. It’s hard to express how many tools and frameworks are written in Python to interoperate with other Python tools. It’s pretty hard to imagine that inertia changing substantially in the next decade.

The Rust projects we have seen above are excellent and will continue to grow for vital and core components, but for them to be helpful for the average data engineer. What was once supposed to be Scala will now be Rust —a backend tooling language to do tasks that need fast and well-maintained code, including a Python wrapper on top.

Writing libraries in Rust feels more like writing long-term infrastructure than writing in higher-level languages such as Python, Java, or the JVM.

What do you think? What is your take on Rust for data engineers?

Resources to Learn More on the Topic

Suppose you want to be up and running within minutes. Karim Jedda has an article, carefully exploring the Rust programming ecosystem as a 10+ years Python developer, checking how to do everyday programming tasks and what the tooling looks like. Shared Services of Canada did a hands-on example with Rust converting raw archive files into JSON for data analysis. Or Mehdi Ouazza’s article where he debates the Battle for Data Engineer’s Favorite Programming Language. 

Learning Rust has many excellent resources. A half-hour to learn Rust, The Rust Book, Rust By Example, Read Rust, or This Week In Rust.

Or Learning Rust with different kinds of formats:

• Are we web yet?
• Are we game yet?
• Are we learning yet?
• Are we GUI yet?
• Are we audio yet?

Or do you want to get hands-on and search for an example project? How about building an Airbyte Delta Lake Destination (Python interface) with delta-rs?

Originally published at Airbyte.com

This post focuses on practical data pipelines with examples from web-scraping real-estates, uploading them to S3 with MinIO, Spark and Delta Lake, adding some Data Science magic with Jupyter Notebooks, ingesting into Data Warehouse Apache Druid, visualising dashboards with Superset and managing everything with Dagster.

The goal is to touch on the common data engineering challenges and using promising new technologies, tools or frameworks, which most of them I wrote about in Business Intelligence meets Data Engineering with Emerging Technologies. As well everything runs on Kubernetes in a scalable fashion but as well locally with Kubernetes on Docker Desktop.

The source-code you can find on practical-data-engineering for the data pipeline or in data-engineering-devops with all it’s details to set things up. Although not all is finished, you can observe the current status of the project on real-estate-project.

What are we building, and why?

A data application that will collect real-estates coupled with Google maps way calculation but potential other macro or microeconomics like taxes, the population of city, schools, public transportation. Enriched with machine learning correlations to know factors that influence the price the most.

Why this project? When I started in 2018, I wanted to find the next best flat to rent but there was no sophisticated real-estate-portal out there in Switzerland. I found this very entertaining post about Datamining a Flat in Munich and wanted to code my own. Presently there are tons of services like PriceHubble in Switzerland or Zillow in the US, but still, it’s worthwhile to optimise for finding your dream apartment or house. On top of it, it is a genuine example that includes enough data engineering challenges.

Starting with web scraping gives you the power to treat every website as a database. We download the latest properties or those who have changed with a change data capture (CDC) mechanism. Zipping and uploading them to S3. With a Delta Lake table, we merge new changes with schema evolution. From there we add some machine learning and data science magic with Jupyter notebooks. Ingest it into the Druid data warehouse. Presenting the data in a business intelligence dashboarding manner. All of it is tied together with a well-suited orchestrator. And of course, everything is running cloud-agnostic anywhere, with Kubernetes. A gist of how the pipeline will look like as of today you see below.

💡 What will you learn?

Below I noted the key learnings which are integrated into a full-fledged data engineering project to illustrate the how part in the most hands-on way. Hopefully, you’ll find something interesting for you!

• Scraping with Beautiful Soup: How to get value out from a website with basic Python skills.
• Change Data Capture (CDC) with Scraping: Using a fingerprint to verify against the data lake if a property needs to be downloaded or not.
• How to use an S3-Gateway / Object Storage: Placing an S3 API in front of your object storage in the so-called “gateway-mode” to stay cloud-agnostic. This allows you to change the object-store from Amazon S3 to Azure blob storage or Google cloud storage with ease.
• UPSERTs and ACID Transactions: Besides schema evolution mentioned above, Delta Lake also provides merge, update and delete directly on your distributed files.
• Automatic Schema Evolution: With the growing popularity of data lakes and ELT, data engineers are left with lots of data but no schemas. To integrate schema and especially schema changes, automatic schema evolution is important.
• Integrating Jupyter Notebooks - the right way: Notebooks hold important data transformations, calculations or machine learning models yet it’s always hard to copy the living code in your data pipelines. We will integrate notebooks as a step of our pipeline with Dagster.
• Learning about Apache Druid: Druid is one of the fastest Data Warehouse / OLAP Solutions. It’s optimized for fast real-time ingestion and immutable data. On the downside, it’s hard to set up, luckily, below you see how to do exactly that.
• Open-Source dashboarding with Apache Superset: How to use Superset with its many out-of-box connections. On top, it’s free of charge compared to Tableau, Looker and others.
• DevOps with Kubernetes: How to run Kubernetes locally and install all of the tools here. If you haven’t used Kubernetes, don’t worry, examples and local set-up with Kubernetes for Docker are included.
• Introduction to features of Dagster: Showing how all of the data engineering parts can be tided together with one open-sourced tool called Dagster (alternative to Airflow).
• And many more which I won’t mention but you’ll hopefully see along the way.

Hands-On with Tech, Tools and Frameworks

In an earlier post about Open-Source Data Warehousing, I focused explicitly on Apache Druid, Airflow and Superset. This post is all about using data engineering in a practical example. To give you an overview of what we use, I extended the tech, tools and frameworks I used in these blog post on top of the newer Databricks Lakehouse Paradigm.

Below you’ll find different chapters for different topics. I included at least one practical example with some hands-on code but kept it minimalistic as the source code is all open in the above-mentioned repositories. Some chapters include extra information or architectural reasoning of why I believed a certain tool or method is especially suited for the use-case. But let’s get started with scraping data implemented with Python.

Getting the Data – Scraping

The internet has an infinite amount of information, that’s why scraping is valuable to know even though less know for data engineers. As a first step, we are getting the properties from a real-estate portal. In my case, I chose a Swiss portal, but you can choose anyone from your country. There are two main Python libraries to achieve this, Scrapy and BeautifulSoup. I used the latter for its simplicity.

My initial goal was to scrape some properties from the web-page by determining how many search page result I get and scrape through each property from each page. While testing around with BeautifulSoup and IPython — IPython is an excellent way to initially test your code — and asking my way through StackOverflow. I found that certain websites provide open APIs which you find in the documentation of their website or with the interactive developer tools (F12) explained below. This will save you from scraping everything manually and therefore also producing less traffic on the site of the provider.

To get started with web-scraping, it helps when you know some basic HTML. To get an overview of the site you would like to scrape your data from, use the above interactive developer tools. You can now inspect in which <table>, <div> or id, class or href your information is found. Most websites with valuable data have ever-changing id’s or classes, which makes it a bit harder to just grab the specific data you need.

Let’s say we want to buy a house in Bern, the capital of Switzerland. We can e.g. use this URL with this search term: https://www.immoscout24.ch/en/house/buy/city-bern?r=7&map=1. R, in this case, is the radius around Bern and map=1 mean we only want properties with a price tag. As mentioned we need to find out how many pages of result we have. We can see this information is at the very bottom. A hacky example that worked for me is I searched all buttons on the page and chose the one smaller and equal three which equals me as the last page number two as of today. An example code to scrape how many pages of search results we have:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

from bs4 import BeautifulSoup
import requests

url = 'https://www.immoscout24.ch/en/house/buy/city-bern?r=7&map=1'

html = requests.get(url)
soup = BeautifulSoup(html.text, "html.parser")
buttons = soup.findAll('button')

p = []
for item in buttons:
    if len(item.text) <= 3 & len(item.text) != 0:
        print(item)
        p.append(item.text)
if p:
    lastPage = int(p.pop())
else:
    lastPage = 1

print(lastPage)

## -- Output --
<button aria-disabled="true" aria-pressed="true" class="bkivry-0 as6woy-0 c2ol4x-0 hXMMbP" disabled="" type="button">1</button>
<button aria-disabled="true" class="bkivry-0 as6woy-0 c2ol4x-0 hXMMbP" disabled="" type="button">2</button>
2

To get to a list of property IDs I assembled a search-link for each search where I grabbed links that stared with “en/d” and had a number in it. Some sample code below:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15

import re
url = 'https://www.immoscout24.ch/en/house/buy/city-bern?pn=1&r=7&se=16&map=1'

ids = []
html = requests.get(url)
soup = BeautifulSoup(html.text, "html.parser")
links = soup.findAll('a', href=True)
hrefs = [item['href'] for item in links]
hrefs_filtered = [href for href in hrefs if href.startswith('/en/d')]
ids += [re.findall('\d+', item)[0] for item in hrefs_filtered]

print(ids)

## -- Output --
['6331937', '6330580', '6329423', '6298722', '6261621', '6311343', '6318070', '6313553', '6317089', '6306531', '6305793', '6296041', '6294327', '6284892', '6283242', '6282624', '6274328', '6251376', '6237199', '6237144', '6231495', '6224144', '6223578', '6209944']

Complete code above you can find on GitHub on solids_scraping.py in functions called list_props_immo24 and cache_properies_from_rest_api.

Storing on S3-MinIO

With an object storage, you provide one single API without lock you in into a cloud vendor and you can always access the same URL/API within your application or pipelines. I use MinIO but there are several others. They normally run on Kubernetes, are open-source and may also boost performance. Plus if you haven't access to your own S3 e.g. locally or on your servers, you can simply create one in three lines of code:

1
2
3
4
5
6
7
8
9

wget https://dl.min.io/server/minio/release/linux-amd64/minio
chmod +x minio
./minio server /data
# — Output —
Endpoint: http://192.168.2.128:9000 http://127.0.0.1:9000
AccessKey: your-key
SecretKey: your-secret
Browser Access:
http://192.168.2.128:9000 http://127.0.0.1:9000

You can access the endpoint 127.0.0.1:9000 programmatically with its key/secrets. On top, you get a full-blown UI as you can see below.

Change Data Capture (CDC)

CDC is a powerful tool and especially in cloud environments with event-driven architectures. I used it to minimize the downloads of already downloaded properties. Besides existing open-source CDC solutions like Debezium, I implemented my own simple logic to detect the changes. Also because I have no access to the source-OLTP database where the properties are stored which you'd need.

I accomplish the CDC by creating two functions. The first one lists all properties to certain search criteria and the second one compares these properties with existing once. How am I doing that? Primarily, I create a fingerprint from each property that will tell me if the one is new or already exstinging. You might ask why I'm not using the unique property-ID? The reasons are I didn't just want to check if I have the property or not. As mentioned in the intro I also wanted to check if the seller lowered the price over time to be able to notify when the seller can't get rid of his house or flat. My fingerprint combines the property-ID and the selling price (called normalized_pricein my data). One more benefit if more columns getting relevant, I could just add them to the fingerprint and my CDC mechanism would be extended without changing any other code.

To have the relevant selling price for each property-ID I will scrape them separately from the website same as the IDs themselves. You can check that code in solid_scraping.py. The function is called list_props_immo24 which returns all properties as a data frame for my search criteria.

The logic for CDC happens in get_changed_or_new_properties in solids_spark_delta.py where I compare the existing once in my delta table with the new coming from list function above. As Delta Lake supports an SQL-API I can use plain SQL to compare the two with this simple SELECT-Statement:

1
2
3
4
5
6

SELECT p.id, p.fingerprint, p.is_prefix, p.rentOrBuy, p.city, p.propertyType, p.radius, p.last_normalized_price
  FROM pd_properties p
  LEFT OUTER JOIN pd_existing_props e
    ON p.id = e.propertyDetails_id
 WHERE p.fingerprint != e.fingerprint
	OR e.fingerprint IS NULL

Adding Database features to S3 – Delta Lake & Spark

To get database alike features on top of your S3 files, you simply need to create a Delta Lake table. For example, to add a dynamic schema to not break ingestions into a data lake or data pipelines downstream is quite a challenge. Delta is doing that and automatically add new columns incrementally in an optimistic concurrent way. As data in a data lake ist mostly distributed files, this is quite hard if you were to do that yourself. But as Delta already enforce schema and stores this information in the transaction log it makes sense to handle this with Delta. In my data sets with 60+ dynamic and changing columns, I made use of this feature extensively.

How to create or read a Delta table then? It can be easily done with providing the format delta opposed to parquet or other formats you'd know:

1
2
3
4
5
6

#create delta table
data = spark.range(0, 5)
data.write.format("delta").save("/tmp/delta-table")
#reading it
df = spark.read.format(“delta”).load(“/tmp/delta-table”)
df.show()

Another feature of Delta is the automatic snapshotting mechanism with time-travel that lets you check older versions of a table. This can become very handy for dimension-tables to protocol history e.g. addresses. This way you can skip a rather complex SCD2 logic. Just make sure to set your retention threshold high enough before you using vacuum (deletion of older data).

1
2
3

#Read older versions of data using time travel
df = spark.read.format(“delta”).option(“versionAsOf”, 0).load(“/tmp/delta-table”)
df.show()

Also handy is it does not matter if you're reading from a stream or batch. Delta support both in a single API and target sink. You can see that well explained at Beyond Lambda: Introducing Delta Architecture or with some code examples. Often used MERGE-statement in SQL can be applied on your distributed files as well with Delta including schema evolution and ACID transaction:

1
2
3
4
5
6
7
8

--A simple example:
MERGE INTO events
USING updates
   ON events.eventId = updates.eventId
 WHEN MATCHED THEN
	  UDATE SET events.data = updates.data
 WHEN NOT MATCHED THEN
 	  INSERT (date, eventId, data) VALUES (date, eventId, data)

Further motivation why I'm using Delta for my project:

• using SQL on top of my distributed files
• simply merge my new properties into my data lake, no need to manually identify data changes
• working with JSONs and each has a totally different schema, I don't need to worry about that with schema evolution
• I get a full-blown transaction log to see what went on
• everything is well compressed and in columnar format stored ready for analytics query with open-source Apache Parquet files
• I have rich APIs in different languages with Scala, Java, Python and SQL
• with deletes integrated I'm prepared for GDPR requirements
• I can always travel back in time to see how the selling price of my properties has risen over time
• no need to worry about size and speed as everything is scalable with Spark, even the metadata.
• future proof with unified batch and streaming source and sink - no need for a lambda architecture where batch and streaming is handled separately
• everything is open-source, the data format with Apache Parquet and Delta Lake itself

To add to the popularity of SQL, I added a Dagster generic solid that can pass any SQL statement along and it will use spark to run on top of my Delta Lake tables. The solid is called _sql_solid (original coming from Dagster airline-demo). The full integrated example in solids_spark_delta.py or below an extract of how I can pass along the merge within Dagster.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

merge_property_delta = sql_solid(
    name="merge_property_delta",
    sql_statement="""
    MERGE INTO {{ target_delta_table }} trg
    USING input_dataframe AS src
       ON trg.propertyDetails_id = src.propertyDetails_id
     WHEN MATCHED THEN
          UPDATE SET *
     WHEN NOT MATCHED THEN
          INSERT *
    """,
    materialization_strategy="delta_table",
    input_defs=[
        InputDefinition("target_delta_table", DeltaCoordinate),
        InputDefinition("input_dataframe", DataFrame),
    ],
)

Machine Learning part – Jupyter Notebook

I'm not a data scientist, still, I wanted to have some insights and fun with my data as well. That's why initially copied one or two Notebook from Kaggle to play around with. In my project, I wanted to integrate them into my pipeline.

Why bother with Jupyter notebooks? Because you most probably have skilled people who are creating the most advanced notebooks with real insight from your data. But unfortunately, these notebooks need to run manually and are not integrated into the data pipelines. There are two options in my opinion. Either you can test and approve notebooks, integrate them into your pipeline, this is basically to copy your python code over into your pipelines. This obviously is a lot of work and does not support changes from the data scientists in the notebooks as these need to be copied over again. So what else could we do?

Good thing there is Papermill that lets you run jupyter notebooks directly. And even better, Dagster integrated Papermill into dagstmill which lets you place one notebook as part of your existing data pipeline. On top of that, you have visibility within Dagster's UI, which lets you open the notebook directly. As well you can interact with the input and output of the notebook or use the output further downstream in your pipeline.

My part of the integration, you can find in data_explorationin solid_jupyter.py.

Ingesting Data Warehouse for low latency – Apache Druid

Most business intelligence solutions include a fast responsive OLAP solution often done with cubes. For example, in Microsoft SQL Server you have Analysis Services. But what should you use if you want an open-source product which is able to handle big data with no problems? One excellent choice is Apache Druid, but if you want to know more details or find other ways for you, check out my blog post about OLAP, and what's next.

Druid is a beast to set up, but luckily in my data-engineering-devops infrastructure project, you find how to set it up on Kubernetes, or locally on your laptop with Docker Desktop which provides a native Kubernetes. Also, check out the original helm chart coming from Druid.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22

#create namespace
kubectl create namespace druid
kubectl get namespaces

#PersistentVolumes (pv) and PersistenVolumeClaims (pvc)
#get context:
kubectl config current-context
#use above context and set namespace to druid:
kubectl config set-context docker-desktop --namespace=druid

#create perstisten volumes
cd $git/data-engineering-devops/src/druid
kubectl apply -f  manifests/base/persistentVolume/volumes.yaml

#druid deployment
cd $git/data-engineering-devops/src/druid
kubectl apply -k manifests/overlays/dev/localhost/sspaeti
kubectl delete -k manifests/overlays/dev/localhost/sspaeti

#Port-forwarding to access druid-UI
kubectl get pod
kubectl port-forward druid-router-86798c8b4c-vjvxj 8888:8888

For the project part, I set it up and ingested some properties, but that was more to test the set-up locally. As speed is not a major requirement for me right now, and Druid eats a lot of resources and hard to run locally, I'm focusing on the Delta Lake to be the single source of thought for my queries.

The UI with Dashboards and more – Apache Superset

No project isn't complete without a nice UI that visualises your insights. For its open-source purposes and features, I use Apache Superset for some time now. Lately, Superset announces version 1.0 and it is among the top 200 projects on GitHub. The founder Maxime Beauchemin and his company Preset are building more and more amazing features, one being to create your own plugins easily.

Superset can easily connect to Druid natively, or it can query a data lake with Delta Lake tables, plus it can handle almost any kind of SQL based database. The dockerfile I use is the original with adding pydruidfor querying Druid. Functionalities as exploring, dashboarding views and how to investigate your data you see below:

Orchestrating everything together – Dagster

Ultimately, the part that glues everything together, the orchestrator. Today there is quite an extended list of orchestrator out there. I tried to highlight the most suitable alternatives to Apache Airflow and went with Dagster for the coming reasons below.

Data pipelines start simple and straight-forward, but often they end up vastly heterogeneous with various APIs, Spark, cloud data warehouse, and multi-cloud-providers. Above a real-live example from GoodEggs which includes mode, networkx, stitch, SQL, Jupyter-notebooks, Slack-connector, cronitor, and many more. This is a complex data pipeline but it is still fairly common to have such an amount of diverse technologies.

Why I'm saying all that? Because this is one place where Dagster shines. It's built with a high-level abstraction in mind, not just as an executor. Even more, you can use different executor e.g. Airflow, Celery, Dask or Dagster itself, no lock-in here. Dagster lets you focus on building your data pipelines. It is made for data-flows and with it to pass data between the solids (their name of tasks). The integration with modes lets you switch from dev to test and production with one click and different resources on each mode. Let's say you won't have a snowflake-db locally available, so you could just mock it or use a simple Postgres locally for testing but with no changing of your data-pipeline code.

You have an elegant way of separating business logic in solids and technical code within resources. Resources are also commonly available in solids and are written once. Meaning your Spark connect, your Snowflake create a table, your rest-call to a certain service, these all can be written once in a resource and all users have it available in every solid.

Dagster provides a beautiful feature-rich UI called Dagit. It includes state-of-the-art GraphQL Interfaces for fetching status, starting, stopping pipelines and many more. As shown in the machine learning part, it closes the boundaries to the machine learning team with the integration of Jupyter notebooks. It's all free and open-source and the team is extremely responsive on both Slack and GitHub.

What about testing? Testing data is very hard and nothing compared to software testing as data is and even tools and framework is dynamic and can change every output of your transformation, as well the size of data changes in dev, test, and production. Dagster's abstraction supports testing profoundly. Type checks and assertions about your data are included. But I'd suggest using the first-class integration of Great Expectation.

On top of that, Dagster embraces the functional programming paradigm. By simply writing Dagster pipelines, you are writing functional solids that are declarative, abstracted, idempotent, type-checked to catch errors early. Dagster also includes simple unit-testing and handy feature to make pipelines and solid testable and maintainable.

All of my examples are implemented with Dagster. Just clone my repo, install Dagster and start Dagit from src/pipelines/real-estate. I’m trying to build an awesome-dagster with common code-blocks as solids, resources and more to be re-used by everyone. Feel free to contribute if you have nice components to add.

DevOps engine – Kubernetes

And finally, the engine everything runs on locally and cloud-agnostic in the cloud is Kubernetes. Quoted from an earlier post in chapter Use a container-orchestration system:

Kubernetes has become the de-facto standard for your cloud-native apps to (auto-) scale-out and to deploy your open-source zoo fast, cloud-provider-independent. No lock-in here. You could use open-shift or OKD. With the latest version, they added the OperatorHub where you can install as of today 182 items with just a few clicks. […] Some more reasons for Kubernetes are the move from infrastructure as code towards infrastructure as data, specifically as YAML. […] Developers quickly write applications that run across multiple operating environments. Costs can be reduced by scaling down […]

To get hands-on with Kubernetes you can install Docker Desktop with Kubernetes included. All of my examples are build on top of it and run on any cloud as well as locally. For a more sophisticated set-up in terms of Apache Spark, I suggest reading the blog post from Data Mechanics about Setting up, Managing & Monitoring Spark on Kubernetes. If your more of a video guy, An introduction to Apache Spark on Kubernetes contains the same content but adds still even on top of it.

Conclusion

We have seen that in order to apply hands-on data engineering methodologies to a real-estate project, you need to know a good amount of the latest big data tools and frameworks. As well as data architecture to assess how these fit together and can be utilised for specific use-cases. I hope I could give you some inspiration and ways to create your own data engineering project. From scraping the web to storing the data in an S3 object store, adding database features onto it, using machine learning capabilities with Jupyter notebooks, ingesting it into a data warehouse, visualise the data with a nice dashboard, connecting everything together with an orchestrator and running it cloud-agnostic.

If you want to test your knowledge, start the Pokemon or Big Data quiz, you will see it’s not that easy 😉. If you like more Open-Source Data Engineering Projects, I curate a list I constantly update.

That’s it for now. If you like the content and want to follow along, make sure you subscribe to my newsletter, check my code on GitHub or visit me on LinkedIn, or Twitter for genuine news about the data ecosystem.

Republished on Medium.

The tools change. The loop doesn’t.

The Invisible Plumbers

When I started my career in 2003, there was no “data engineering”. There was no big data, no data science. We called it Business Intelligence. Data Warehouse Developer. ETL Developer.

We were the plumbers of the organization. And like plumbers, nobody noticed us until something broke.

Being a data engineer means: you’re building the foundation that everyone stands on, but when the presentation goes well, the data scientist, the app developer, anyone who presents gets the applause. When the executive makes the right decision, the analyst gets the credit. When the dashboard loads in 1 second instead of 20, nobody says anything at all.

But when one number is wrong? When a pipeline is 10 minutes late? When someone asks for “a small change” and you explain it’ll take a day, or a week to fix it?

That’s when everyone notices you. And shares their opinion on how to make it better.

“Why does this take so long? It’s just one column. Why isn’t it real-time?”

They don’t see the 147 downstream dependencies. The three systems that need a fuzzy-logic join. Or the security measures that go through three different subnetworks. The backfill that’ll take 6 hours to run. The schema that hasn’t been touched since 2021 because the last person who understood it left the company long ago.

This is the paradox of data engineering: when you do your job, you’re invisible. When anything goes wrong, you’re under a microscope.

The Epochs: A 50-Year Journey

To understand where we are today, you need to understand where we came from.

1970s: The Beginning

Edgar F. Codd proposed [[SQL]] in 1970. A way to abstract the complexities of data storage. By the 1980s, it became the standard. IBM built System R. Oracle launched their RDBMS in 1979.

The foundation was laid. But nobody called it “data engineering” yet.

1980s-1990s: The Warehouse Era

[[Bill Inmon]] formalized data warehousing principles in the 1980s. Many call him the father of data warehousing. Then in 1996, Ralph Kimball published “[[The Data Warehouse Toolkit (Ralph Kimball)|The Data Warehouse Toolkit]]” and gifted us with [[dimensional modeling]]—star schemas, fact tables, slowly changing dimensions.

These concepts? They’re still relevant today.

2000s: When “Big” Changed Everything

The dot-com bubble burst. Tech titans were born such as Google, Amazon, Yahoo, hitting walls their databases couldn’t scale past.

So Google released two [[Data Engineering Whitepapers|groundbreaking papers]]: the Google File System in 2003, MapReduce in 2004. Yahoo responded with Hadoop in 2006. Hardware prices plummeted.

Suddenly, we weren’t just BI engineers anymore. We were “Big Data Engineers”. We had to know traditional relational databases AND the new open-source filesystems. The skillset kept expanding—from data modeling to software development to mastering Hive and Spark, all coordinated with R and Python.

The term “big” was everywhere. But how big is “big”? Nobody really knew. We just knew the old ways weren’t working anymore. And Facebook and co showed us the way.

2010s: The Cloud Changes the Game

Amazon announced AWS. Google Cloud and Azure followed. Companies no longer needed to own hardware. The flexibility was unprecedented, and we could get any DWH on demand.

Redshift. Snowflake. And then the open-source wave hit:

• Airflow for orchestration (2014)
• Superset for visualization (2015)
• dbt for transformation (2016)

And in 2017, Maxime Beauchemin—after creating both Airflow and Superset—published “The Rise of the Data Engineer”. He defined, for the first time, what data engineering actually meant. He explained the shift from business intelligence to data engineering.

I remember releasing my first viral article in March 2018: “Data Engineering, the future of Data Warehousing?” It got 200 likes. Back then, that was a lot 😉.

Since then? New technologies appeared weekly. The [[Modern Data Stack]] was born.

2020s: DevOps Meets Data Engineering

This is where it gets interesting.

Data engineering isn’t just about moving data anymore. It’s about infrastructure as code, version control for data, CI/CD pipelines, Kubernetes, Docker, and Terraform.

The skills needed have exploded. You need to know:

• SQL (still the foundation)
• Python or Scala
• Cloud infrastructure (AWS/GCP/Azure)
• Linux and bash scripting
• Git for version control
• Data modeling (the lost art)
• Business logic (the most important)

DevOps principles are now [[The State of DevOps in Data Engineering|table stakes]]. You’re not just building pipelines. You’re building systems that need to self-heal, auto-scale, and deploy without downtime on any environment.

And today? AI agents? They’re the latest chapter. But under all the hype is the same eternal truth: you need fresh, organized, clean data.

The Eternal Loop: Same Problems, New Tools

Here’s the uncomfortable truth: we’ve been solving the same problems for 50 years.

In 2005, we had SSIS and star schemas. “The cube is rebuilding” was the pain point.

In 2015, we had Hadoop and Spark. “The cluster is full” was the nightmare.

In 2025, we have dbt and Snowflake. “The bill is how much?” is the new horror story.

The tools change. The problems don’t.

Last month I analyzed a 200-line dbt model as part of a larger GitHub repository. You know what it was doing? Exactly what we did in 2005 with stored procedures. Same business logic. Different syntax. I laughed. Then I cried a little. (just kidding, I didn’t 😆)

An old data warehouse architect from 2003 once drew a star schema on a whiteboard in 40 seconds. It would take my team three sprints to model in Oracle Warehouse Builder (OWB). He said, “We called it just another day at the office”.

We’re not really any smarter than the people before us. We just have better marketing 😉.

What Actually Matters (And What Doesn’t)

Here’s what I’ve learned after 20+ years.

The Excel File That Saved Me

I was in a coffee meeting with a finance analyst—call her Maria. Fifteen years at the company. She opened her laptop and showed me an Excel file (sometimes it was Microsoft Access DB with a custom UI!).

Forty-seven tabs. Formulas referencing other files on a shared drive. VBA macros from 2012. VLOOKUP nested inside SUMIF.

“This is how we calculate the quarterly forecast”.

From my perspective of making it available to everyone and needing to understand it, I was a little shocked. I’d spent three weeks reverse-engineering the business logic, trying to understand it, trying to recreate it in SQL Server, and adding it to our data warehouse. But the numbers were never the same, close most of the time.

After having multiple such experiences, sometimes Microsoft Access databases with custom UI built in (!!!), I learned something. Though my initial reaction was shock and horror, I learned that Excel isn’t the enemy. Excel is the business telling you what they actually need.

When someone asks to export to Excel, they’re not rejecting your work. They’re telling you something. Maybe your dashboard is too slow. Maybe they need to add a column you didn’t think of. Maybe they just need to feel in control of their analysis.

Power users will overengineer everything, but ask them for the Excel file and you might get validated business logic and ETL code for free. Win-win.

The Real-Time Lie

Everyone wants real-time. “We need to see this data instantly”, they say. “For decision-making”.

I always ask: “What decision will you make differently if you see it 10 minutes sooner?”

Most of the time, they can’t answer. There’s a small percentage that needs it: air traffic control, fraud detection, Black Friday e-commerce. But the rest? They just think real-time serves them better.

Real-time adds a much higher complexity. Harder debugging. Harder backfills. Harder historization. The question is, for what? So someone can watch a number update every 30 seconds instead of every hour?

Push back on “real-time”. Start with hourly refreshes. It’s almost always enough.

The Schema Change: A People Problem

They said it was small. Just renaming user_id to customer_id.

You trace the lineage. 147 downstream dependencies. Three teams. One undocumented view from 2019 that somehow powers the CEO’s dashboard.

That’s when you realize: schema changes are usually people problems, not technology problems. The reason things break is when upstream producers don’t own responsibility for downstream analytics and don’t communicate the changes. There’s no process in place. Just assumptions.

Fix the people process first, then update the code.

The Lost Art of Data Modeling

Max Beauchemin once said in an interview: “I like the analysis side. I think I’m a good data modeler. It’s kind of a lost art, so I still do a lot of our data pipelines”.

He’s right.

After years of “just dump it in the data lake”, people are rediscovering that structure matters. Data modeling forces you to think about:

• [[Granularity|Grain]]: What’s the lowest level of detail we need for this data?
• [[Entity Relationship Diagram (ERD)|Relationships]]: How do these entities connect?
• [[The Goal of Business Intelligence|What the business needs]]: What user insights they cannot know today, but lie in the source system provided, or combined with other data sources.

It’s the difference between a data warehouse and a data dump.

But here’s the thing: I believe AI will bring us back to the fundamentals. When AI-generated code breaks and you’re out of context, what then? That’s where the fundamentals save you. Data modeling. Understanding the grain. Knowing SQL deeply, not superficially¹.

Someone needs to understand and refactor generated code. Someone needs to simplify. That someone is you.

The Lost Code We Inherit

You’ll inherit code from someone who left. Everyone does.

I once found a DAG called final_v3_FIXED_REAL_FINAL.py. Inside was a comment:

1

# Mike: I don't know why this works. Just leave it

Mike was right. I left it.

The biggest pitfall? Trying to recreate everything to your taste. Accept the legacy. Adapt or improve one thing at a time. The motto “Don’t touch what works today” really applies to legacy code most often.

Usually, the previous engineer wasn’t naive or stupid. They were solving different problems with different constraints. Your job isn’t to make it beautiful (sometimes it helps!). Your job is to keep it running while slowly making it better over time.

The Books That Actually Matter

As the cycles come and go, these books helped me throughout the cycle², and can be applied to this day.

Designing Data-Intensive Applications by Martin Kleppmann about distributed systems and how to build them. Wait a little, version two is just around the corner.

[[The Data Warehouse Toolkit (Ralph Kimball)|The Data Warehouse Toolkit]] by Ralph Kimball. Someone in 2045 will still need to understand fact tables and dimensional tables.

Fundamentals of Data Engineering by Joe Reis and Matt Housley. A great start to know about all the concepts and principles you hear everywhere, including in this article.

Patterns of Data Engineering (PoDE) by me. If you want, you can also read my unfinished online book, which starts with the state of the art, the history and key convergent evolution in data engineering, about the ever-returning cycle we talk about here, and explains them with higher-level patterns.

The first two books don’t mention Snowflake, Lakehouse or dbt. They mention problems that existed in 1995 and will exist in 2045. That’s the Lindy Effect, and how you know they’re worth reading.

What I Know Now That I Wish I Knew Then

If I could go back to 2003 and talk to my younger self, here’s what I might say. Boy:

1. The tools will change. The fundamentals won’t.

Stop chasing every new framework. Learn [[data modeling]]. Learn how data is flowing. Learn SQL deeply, not superficially. Learn how humans make decisions. Everything else is syntax.

In 2026, AI helps us write code faster. But someone still needs to understand the fundamentals, the [[Data Engineering Lifecycle]]. That someone can be you.

2. Talk to the business people.

This is a crucial lesson in my journey. What you’ll learn from them will make you inevitably a better data engineer. Technical skills can be learned, outsourced, automated. Knowledge about the business is much harder.

The best data engineers aren’t the ones who know every new tool. They’re the ones who know why the data matters.

3. You’re building the foundation, not the showcase.

When the presentation goes well, the data scientist, the AI engineer gets the credits. When the executive makes the right decision, the analyst gets credit. When the dashboard loads fast, nobody says anything.

But when one number is wrong? Everyone sees you.

Accept this. You’re a plumber. Be the best plumber in the world and make sure nobody ever thinks about the pipes.

4. Data quality is learned through pain.

You can’t understand data quality from a textbook. You need to see bad data. If you start looking, it won’t take long, and you’ll see really bad, production-breaking data. That will teach you what good data looks like.

And you’ll only get faster by talking to the people who use it.

5. Presentation matters more than you think.

No matter how fancy your pipeline, how elegant your code, how profound your insights—if the presentation isn’t right or the data quality is terrible, no one cares.

Throughout my career, presenting data understandably has been as important as building the pipeline. That’s why these days, I focus on the storyline and craft extensively.

6. Set boundaries early.

This job will take everything you give it. The people who succeed aren’t the ones who work 80-hour weeks. Sure, in the beginning you need it here and there too. But over time, you need to learn to [[Hell Yeah or No|say no]]. Document things so you can take vacation. Build systems that don’t require you to be online at 3 AM.

Future you will thank you.

7. Don’t chase every trend.

Data engineering is still going strong. Stronger than ever. AI won’t take our jobs any time soon. The opposite is true. There will be more chaos, and people who know how to model data, understand business requirements, and deliver high-quality insights will always be needed.

Plus, every AI solution out there needs data, a lot of data, and probably a plumber to fix the pipeline too. Use the knowledge of past years building data pipelines. We don’t need to rebuild everything every 5 years.

The Loop Continues

It’s 2026. I’m building a pipeline with DuckDB and Rill. The business wants faster dashboards and better insights. They want to edit data themselves. They want to use an AI chatbot. Or sometimes they just rename a column in the source system without telling anyone.

Here we go again.

But here’s the thing: I still love it. Especially when I can write about the learnings.

I don’t miss the late nights or the schema changes or the never-ending rewrites. I love the moment when you finally get the data right and someone in finance sees something they’ve never seen before. When a dashboard actually changes a decision. When the CEO asks a question and you can answer it with data.

That’s the job. Not the tools. Not the frameworks. Not the buzzwords.

The moment when data helps a human make a better decision.

The Final Truth

The tools will change. The vendors will rise and fall. Snowflake will be replaced by something else. The latest new shiny tool will become the legacy tomorrow. AI agents will be the next big thing, and then something after that.

But someone, somewhere, will always need to:

• Understand the grain of a business
• Know why the numbers don’t match
• Explain to the CEO that the data they want doesn’t exist yet
• Debug why a pipeline broke at 2 AM
• Figure out why production data looks different from dev data

That someone is you.

You’re the invisible plumber. The unsung engineer. The person who makes sure the foundation doesn’t crumble while everyone else builds on top of it.

And honestly? It’s a pretty damn good job if you like to work quietly, helping a large part of the business.

Because 50 years from now, when we’re using tools we can’t even imagine today, someone will still be ingesting data, modeling it, transforming it, serving it. Someone will ask for a change. Something will break.

The loop continues. The problems remain. Only the tools change.

And that’s okay. Isn’t that somehow beautiful? Because beneath all the hype, all the new frameworks, all the promises of “this time it’s different”—there’s you, the data engineer 😉. Understanding the data. Knowing the business. Building the foundation.

That’s [[Data Engineering]].

[!tip] Inspiration

This piece was inspired by the confessional storytelling style of Diary of a CEO. If you enjoyed this format applied to data engineering, let me know—I’d love to hear your own stories from the field.

After covering the essential toolkit in Part 1 (essential tools for your machine) and Part 2 (infrastructure and DevOps), this article teaches you how and in what order to learn these skills. The roadmap provides a structured path to level up during the slower summer months.

The roadmap is organized into 3 weeks that you can learn at your own pace and time availability:

• Week 1: Foundation (SQL, Git, Linux basics)
• Week 2: Core Engineering (Python, Cloud, Data Modeling)
• Week 3: Advanced Topics (Streaming, Data Quality, DevOps)

![[de-roadmap.webp]]

How to use this guide: Each section contains curated resources (articles, videos, tutorials) for that topic. Click on the links that interest you most. It’s meant as a guided roadmap to learn the fundamentals of a “full stack” data engineer.

[!tip] Learning at Your Own Pace

While structured as a three-week program, everyone learns differently. Pick what’s most relevant to your goals and skip sections you won’t need immediately or in the near-term future. Consistency matters more than speed. Sometimes we forget how far 30 minutes a day can take us. And no, after three weeks, you won’t know everything you need to know, but you’ll be able to understand the problems and identify potential angles to solve them.

Week 1: Foundation and Core Skills

Let’s get started with building your technical foundation skills for data engineering.

You can learn the foundational skills in many ways: there are bootcamps, courses, blogs, YouTube videos, hands-on projects, and many more ways to learn them (free and paid ones), including the more advanced skills.

SQL Foundations

Probably the most important skill of any data engineer, at any level, whether they are closer to the business or more technical, is SQL—the language of data. You can descriptively explain what you want from your data much more precisely than natural language through LLM workflows. That’s why it will always be a core skill. For example, in the English language, you won’t specify the partitions or the exact date range (including or excluding the current month). There are many questions that you need to define in your WHERE statement or in the SELECT, which you would miss otherwise.

To get started with SQL until you master it, you can follow this roadmap below:

• Start with understanding SQL.
• Database design principles, from relational database basics to key concepts for beginners. Learn DDL (ALTER, CREATE), DML (INSERT, UPDATE, DELETE), and relational theory by Edgar F. Codd, who invented the theoretical basis for relational databases.
• Advanced SQL queries, such as Window functions for performing advanced aggregations without additional subqueries within the current query. Or, CTEs are a powerful syntax that allows for better readability, creating aliases for sub-queries, and even recursion is possible.
• ACID properties and transactions within databases such as Postgres, MySQL, and DuckDB.
• Learn the differences between OLTP vs. OLAP with a beginner’s guide. Also, check out an explainer of What is OLAP?
• dbt core and SQLMesh: frameworks to encapsulate SQL into a structure that can be versioned, tested, and run in order, including well-documented lineage as a web page.

Version Control

If you use SQL, very quickly you’ll want to work with coworkers and want to version it so as not to lose essential changes or to roll back added bugs.

Therefore, you need version control. This short chapter gives you some starting points for the most common one.

• What is version control - a visual guide to version control.
• The tool, Git fundamentals.
• GitHub/GitLab Collaboration: Learn about platforms like GitHub and GitLab for hosting Git repositories and for sharing and collaborating with others. Main features include Pull Requests and Issues for communicating your changes in a structured way.
• Learn the different git workflows. Also, check out git worktree. Although it’s a bit advanced, it’s good to know it’s there, especially if you need to work on different branches simultaneously without constantly stashing or committing your unfinished changes before switching to another branch.

There are many more helpful topics, such as GitHub Actions/Pipelines for CI/CD or basic automation (uploading documents to a website, checking grammar automatically before publishing, etc.). However, for the first week, let’s keep it simple and move on to the next chapter: Linux and scripting.

Environment Setup, Linux Fundamentals & Basic Scripting

Set up your development environment and master essential Linux skills for data engineering. This depends on your operating system of choice, too, but most data engineering tasks are typically run on servers. In almost all cases, they are executed on Unix-based systems. That’s why Linux fundamentals are key to elevating your data engineering skills.

Below are the resources and roadmap to learn about these topics:

• Bash scripting essentials, starting with the basics of bash scripting, including variables, commands, inputs/outputs, and debugging. Alternatively, use this course with an interactive command line in the browser: Linux command line basics (Paid).
• Package managers (Apt, yum, Homebrew, Wget): How to Use Package Managers in Linux? (APT and YUM) and Homebrew for macOS
• SSH and remote connections: Connecting to a remote server and fixing a DAG or updating a script on the fly.
• Development environment setup: Simple yet powerful dev setups: MacOS setup with pyenv, docker, uv, VSCode, Linux (Omakub, Omarchy) and Windows Setup for data scientist.
• Cron jobs and scheduling: Basic automation scripts without the need for a heavy tool.

Congratulations, this wraps up week one. If you have watched, experimented, and taken notes, you now possess the fundamentals of data engineering and, frankly, any engineering or technical job. Give yourself some time to ponder and review, and then proceed to week two below.

Week 2: Core Data Engineering

Week two is all about the essential data concepts, primarily established principles for manipulating and architecting data flows for data engineering tasks.

Data Modeling & Warehousing

To avoid creating independent SQL queries and persistent data tables without connected data sets, we need to model our data with a more holistic approach.

This is where the concepts of so-called data modeling and the long-standing term data warehousing originate. The sole purpose of these is to organize data optimized for consumption, whereas data in Postgres and other operational databases is optimized for storage.

This chapter will teach you and point you to key knowledge to prepare you to model enterprise workloads.

• Data modeling is a significant one, and somewhat underappreciated these days. However, with the rise of AI and automation, it hasn’t been more critical to learn.
  • Dimensional modeling with a star schema.
  • Snowflake schema vs star schema: Understanding when to use normalized vs denormalized dimension tables.
  • Data normalization: 1NF, 2NF, 3NF principles for reducing data redundancy
  • Fact tables vs dimension tables: Understanding measures, metrics, and descriptive attributes.
  • Granularity is a key concept to understand, so your facts will not suffer from too low detail that is slow, or too high-level detail that loses crucial information when drilling down in a dashboard.
• Data warehouse design methodologies:
  • Kimball methodology: Bottom-up, business process-focused approach.
  • Inmon methodology: Top-down, enterprise data model approach.
  • Data Vault 2.0: An approach with hubs, links, and satellites for agility and scalability.
• Advanced modeling concepts:
  • Slowly changing dimensions: Handling changes in dimension data over time.
  • Bridge tables and many-to-many relationships: Managing complex relationships in dimensional models.

Python for Data Engineering & Workflow Orchestration

After SQL, Python is the next most important language to learn. While it’s beneficial to have deep knowledge about SQL, and you only need preliminary Linux skills to get around a server and run some commands from the command line, Python is the utility language of data. It’s the glue code that connects everything you can’t achieve with SQL, most notably working with external systems and orchestrating your data workflows with Python libraries and frameworks.

Orchestration and other more modern tools help you automate and organize, as well as version your data tasks and pipelines.

• Starting with a Python general introduction.
• DataFrame and data manipulation with Pandas, Polars and DuckDB. Navigating the Dataframe Landscape and DuckDB vs Pandas vs Polars for Python Developers, Video Format
• Python libraries for Data validation with Pydantic or Data Testing with pytest.
• Utilitarian Python knowledge. Connecting to any API quickly with FastAPI.
• Workflow orchestration is almost as important as the Python language itself. Apache Airflow is the biggest name. You learn about task dependencies and scheduling, as well as how orchestration and integration of data tools and stacks work through workflow management. Also, check out related DAG design patterns for guidance on designing pipelines that are easy to maintain and separate business logic from technical logic in an organized and conventional manner.

[!example] Example Data Sets to Test for Yourself

To manipulate data or create an example project, you can use the provided datasets out of the box with DuckDB: Example Datasets, containing interesting datasets such as HackerNews, Foursquare, PyData, StackOverflow, and many more.

Cloud Platforms Introduction

Getting to know major cloud platform providers can save you a significant amount of time and enhance your employability because you know how to work around permissions, the services provided, and how to automate specific tasks. Ensure you select the right provider based on your location and primary use, or the company you prefer to work for.

• Introduction to AWS, Azure, or Google Cloud. Vital is permission management, such as security and IAM basics, on all platforms.
• Dedicated data services: MotherDuck, BigQuery, Fabric, hosted Airflow (Azure & AWS).
• Object Storage or blob storage setup on all platforms.

Depending on where your resume positions you, you’ll do different work. But some sort of analytics through business intelligence (BI) is always involved. Visualizing your data and showing it in a way that makes sense immediately is hard; that’s where BI tools and data visualization come into play.

• Introduction to BI tools and using notebooks. Others are Jupyter Notebooks, Hex, DeepNote, and many more. Check Jupyter notebooks for analytics, which is a super helpful toolkit for data analysis and iteration.
• Metrics and KPI design with metrics layers and semantics.
• Data visualization best practices. Tools like color management and a high-level grammar of interactive graphics help understand data presentation. Hichert SUCCESS Rules is another great option, although it is only available in German. Check also Data Visualization with Hex/Preset and DuckDB/MotherDuck.
• Self-service analytics enables business people to serve themselves.

This concludes Week Two. You’re ready to tackle the advanced topics in Week three.

Week 3: Advanced Topics

This final week focuses on advanced topics, including data quality and streaming. This last part of the data engineering roadmap focuses on cost optimization, data quality, event-driven approaches, DevOps learnings, and advanced data quality and observability.

Some of these topics are rarer approaches and should be avoided initially, but there’s a time when you need any of them.

Stream Processing & Event-Driven Data

Event-driven approaches or integrating your data as a stream, end-to-end from source to your analytics, is sometimes a must and business-critical, especially for ad-tech or sports, where you need live results that are as up-to-date as possible.

Understanding stream processing fundamentals is especially beneficial for validating users’ requests for real-time data insights, as they will often ask for it, but it’s not always necessary.

• Event-driven architecture and design practices: How do they differ from batch loads? Key players in this category are Apache Kafka and Flink.
• Real-time analytics patterns: Change Data Capture (CDC) and the difference in propagating that stream compared to batch. See Postgres change data capture possibilities.

Data Quality & Testing

Implementing robust data quality frameworks and testing strategies is crucial for maintaining a stable data platform. Most often, it’s quick to set up a data platform, or a stack to extract analytics from your data, but doing it stably and with high data quality is an entirely different job. The tools in this chapter will help you with that.

• Great Expectations and other data quality frameworks.
• Unit testing for data pipelines: How to test your data and pipelines in an automated fashion.
• Data lineage and governance: How to get the lineage of your data flow.
• A Beginner’s Guide for Observability. Be sure to learn about Data Contracts, a concept for defining data interfaces between data and business teams.
• Metadata Management: Data discovery with data catalogs, ratings of datasets to know which ones are actively used and of good quality. Check also the Schema registry management to handle metadata.

Cost Optimization & Resource Management

Most of the time, especially if you use cloud solutions, the price to pay for these services is relatively high. Therefore, stopping the creation of a heavy temp table on an hourly basis can save a significant amount of costs. Consequently, it’s crucial to debug heavy SQL queries or wasted orchestration tasks, including orphaned ones that aren’t connected to any upstream datasets or that aren’t in use.

Stacks that don’t run in the cloud are optimized differently. Here, you don’t pay for cloud services, but to run your own. That’s why you optimize for team members and tasks. As data engineering tasks are elaborate, spending time on the right tasks can save a lot of money, too.

In the past, it was referred to as performance tuning. At that time, we were optimizing for speed, which remains the case today. Similarly, if you maximize performance, you also improve cost efficiency at the same time, as it runs for shorter periods. Over time, this can result in significant savings.

• Cloud cost monitoring and optimization: Tools to monitor the cost and usage of data engineering tasks.
• Performance Tuning: Indexing, partitioning strategies, and caching mechanisms are important components, as is query optimization for better efficiency and lower cost.
• Storage tiering and lifecycle management

Infrastructure as Code & DevOps

Infrastructure management and deploying new software in an automated fashion typically occurs through Infrastructure as Code (IaC) using Kubernetes or a similar platform. That’s why it’s good to have preliminary knowledge about these tools and when to use them.

• Docker containerization is a good start; here’s a beginner’s guide.
• Kubernetes and Terraform basics.
• Monitoring and logging explained.
• Advanced CI/CD for deploying entire data stacks and data platforms.

That’s it. This is a three-week roadmap with numerous courses and links to help you learn data engineering. Let’s take a break and dive into the final part, observing what we’ve learned throughout these three weeks.

Congratulations, You’ve Learned the Essentials of Data Engineering

This roadmap provides the foundation, but data engineering is a field that requires continuous learning. Stay curious, build projects, and connect with the community. The skills you’ve developed here will serve as your starting point into more specialized areas as you grow in your career.

A quick recap of what you have learned. By the end of this 3-week roadmap, you should have learned a lot, especially the key components of data engineering. With a little bit of picking and choosing, it should have been fun to engage in new, interesting, and potentially unknown topics.

By Week 1, you learned how to write SQL to query the data you want, and some additional functions that SQL provides that you didn’t know before. You know how to safely version control your SQL statements and collaborate with others on them. And you have some basic Linux skills.

After Week 2, you can navigate and use a cloud-based data warehouse on one of the major cloud providers of your choice. You learned different ways to model your data and its flow, as well as which Python libraries and helper frameworks are available.

Week 3 enables you to understand basic analytics skills and present data to clients. You know how to implement the glue code between SQL and run it on Linux using workflow orchestration tools. You have a rough idea of what real-time data workloads look like and how they differ from batch workloads. You should have an understanding of how to package production-ready code for deploying scalable data stacks using DevOps tools and methodologies. You have heard and seen various approaches to architecting an enterprise data platform.

What’s Next?

All of it will help you build your portfolio and land your dream data engineering role. Each week builds upon the previous, creating a comprehensive learning experience that mirrors real-world data engineering challenges.

Throughout the entire process, it’s beneficial to build your online portfolio, where you showcase your data engineering learnings, Git projects, website, and links to hackathons you participated in, among other things that demonstrate your motivation. Above all, sharing is also fun; people will reach out to you after reading your content, especially if they learn from it too.

Remember to take your time learning new concepts. If you give yourself time to digest, you learn more easily, you’ll be able to recall specific terms better, and it’s easier to connect the knowledge—this is how our brains learn.

Consistency is key. Dedicate 1-2 hours daily for a couple of weeks, and you’ll be amazed at what compounding and consistent learning can achieve.

I hope you enjoyed this write-up. If so, you may also find the essential toolkit article for data engineers, available in Part 1 and Part 2, or check an End-To-End Data Engineering Project with Python and DuckDB.

If you want more? Check out the Mastering Essentials resources by MotherDuck, or follow their YouTube channel for additional resources. If you like DuckDB and need a cost-efficient data warehouse or data engine, check out MotherDuck for free.

Further in-depth content can be found and learned through bootcamps, events, and courses. Please don’t give up; it’s a lot to take in when you start. Begin with the fundamentals as guided in this roadmap, and also follow your interests. It’s better to learn something that might not be suitable right now, but because you are passionate about it, learning comes much more easily. And over time, that knowledge may be put to use at a crucial moment later on.

Full article published at MotherDuck.com - written as part of my services

I’d argue that today’s data engineers face similar challenges, but with the added complexity of infrastructure setup. We’re architects of entire data ecosystems, orchestrating everything from real-time pipelines to AI workflows. The secret? Infrastructure as Code and DevOps principles that transform scattered server management into elegant, declarative configurations.

The catch is that while abstractions have made complex deployments more accessible, the toolkit has exploded in scope. One day, you’re optimizing SQL queries, the next, you’re debugging Kubernetes deployments, and by afternoon, you’ll be explaining data quality metrics to stakeholders who just want to know why their dashboard is empty.

This is Part 2 of my in-depth exploration of the modern data engineer’s toolkit. While Part 1 covered the fundamentals of your development environment, programming languages, and core productivity tools, this essay addresses the more advanced technologies—such as data processing, infrastructure, data quality, and observability—required to transform data pipelines into production-grade data platforms.

We’ll explore everything from SQL engines and workflow orchestration that form your daily toolkit to DevOps practices that make your deployments bulletproof, and the advanced utility tools that help you sleep better at night. Additionally, we’ll explore the soft skills that can make the difference between a data engineer and a data engineering leader.

Data Processing and Analytics

Continuing from the developer productivity and data engineering programming languages discussed in Part I, we have data processing and analytics technologies that are at the core of data engineering. SQL, relational databases, and BI tools are the bread and butter of everyday work, and Python is the glue language that ties everything together.

But most of the time, we must also set up a project that connects all the dots through orchestration, whether it’s a simple cron job or Python script.

SQL and Databases

SQL is the language of data. SQL is a fundamental skill for doing any data work. There’s almost nothing you do without needing SQL. If you work with a REST API with no direct SQL interface, it’s still beneficial to know, as the REST service will most certainly perform a SQL query against the database based on your REST request.

With that said, what SQL engines and databases do data engineers use?

The most common relational databases are also called OLTP databases:

• SQLite: A single-file database that is very handy for web development or when you need a database that can go with the code to avoid long latency for network or fetching and pushing data.
• Postgres: Perfect for any transactional and smallish data, but also scales up relatively high.
• MySQL / MariaDB: Wide adoption before Postgres, good performance. MariaDB forked from MySQL around the Oracle acquisition of MySQL (acquired through the Sun purchase).

Analytical databases that speak SQL - also called OLAP - are optimized for fast query responses:

• DuckDB: A single-file OLAP database, optimized for analytical queries.
• MotherDuck: Scaled out DuckDB in the cloud, DWH in minutes.
• ClickHouse: A fast analytical (OLAP) database.
• StarRocks: A newer fast analytical database, focusing on making data-intensive real-time analytics easy.
• Cloud Data Warehouses: Snowflake, BigQuery, Redshift, Azure Fabric

Database utilities that help us with both:

• pg_duckdb: small library and plugin to make Postgres work with DuckDB, mainly extending Postgres with analytical features.
• JDBC / ODBC and newer versions Arrow-ADBC.
• Apache Calcite: SQL parser and query optimization framework

Python Processing Tools

Python, on the other hand, is the ultimate toolkit language. Pulling data from a REST API or web, cleaning out some insufficient data, and storing it in Postgres. How would you do that in a safe, ordered fashion? Right, Python. It allows you to easily reach an API and automate tasks that Bash can’t handle.

Besides the generic Python libraries in Part 1, here are Python data processing libraries, potentially lesser-known, and suitable for advanced use-cases:

• Ibis: It provides a lightweight, universal interface for data wrangling, helping explore and transform data of any size, stored anywhere.
• Dask: A flexible library for parallel computing. Dask scales Python code from multi-core local machines to large distributed clusters in the cloud.
• ConnectorX: The fastest library to load data from the database to DataFrames.
• Modal: A cloud function platform that lets you run any code remotely within seconds.
• reladiff (formerly data-diff by Datafold): Tool to efficiently diff rows across databases
• Quokka: An open-source push-based vectorized query engine.
• Vaex: High-performance library for lazy out-of-core DataFrames, to visualize and explore big tabular datasets.
• Xorq: A declarative framework for building multi-engine computations.
• gspread: Work with Google Sheets through Python API, or with DuckDB.

Want more? Check out the Awesome Python List with thousands of more frameworks, libraries, software, and resources.

Workflow Orchestration Platforms

A key tool, often used within Python, are data orchestrators. These orchestrate the workflow of data processes in certain needed steps.

These are typically in Python, such as Apache Airflow, Dagster, Prefect. But there are also others, such as Temporal, Kestra, Mage, Argo Workflows, Flyte, and many more.

Analytics and BI

Besides relational databases, SQL, and Python, in all cases, you want to present the data to your users or stakeholders. This is where BI tools, Notebooks, and data apps for visualization come into play.

There’s plenty out there, but here are the major ones and my favorites:

• Apache Superset: Original open-source BI tool.
• Rill: Open-source and BI-as-Code platform.
• PowerBI: Microsoft’s business intelligence platform.
• Omni: Business intelligence platform that helps companies explore data with a point-and-click UI, spreadsheets, AI, or SQL.
• Sigma Computing: Next-generation analytics and business intelligence platform with SQL in a familiar spreadsheet interface.
• Lightdash: Instantly turn your dbt project into a full-stack BI platform.
• Tableau: An Enterprise BI tool that has existed for a long time, with powerful ETL and other features.
• TARGIT: Enterprise BI solution specializing in industry-specific implementations in the Nordics.

Beyond BI tools, there are also notebooks:

• Jupyter Notebook / Zeppelin, Marimo: Open-Source notebooks
• Hex, Deepnote, MotherDuck Notebook: Closed-source
• More exotic ones: Count (canva style), Quadratic (spreadsheet style), Excel (mother of BI tools)

DevOps and Infrastructure

Once you have a setup with integration, orchestration, and visualization, you usually need to scale it or deploy it to internal cloud servers or one of the major cloud providers. You typically use something more than plain Docker Compose or a quick uv init for setting up all relevant Python settings. Usually, it involves Kubernetes, Terraform, or Infrastructure as Code.

Either you pay for a service to do that for you, or if you have chosen a set of open-source tools, you mostly end up doing it yourself.

Popular frameworks, such as Terraform, Helm, and Ansible, as well as other scripts, can be deployed on any cloud. Typically, a Kubernetes cluster is used to deploy them. It’s the de facto standard for cloud-agnostic deployment and works well for data engineering projects, as you declaratively define the state you’d like to have for your data platform. Kubernetes matches that and ramps the right amount of server, CPU, memory, etc., to make it runnable and scalable on any cloud.

Most of the time, it includes setting up an automated CI/CD pipeline that handles automated testing, deployment, version control, and all the software engineering best practices for data engineering.

Infrastructure as Code, GitOps, and DataOps

DevOps has become a bigger part of data engineers’ work in most scenarios in recent years, making deployment of every updated OSS tool straightforward, easy to test, and reproducible.

Making the data stack modular so that additional tools can be added with a clearly defined path for integration, such as metadata, logging at the same place, and security, so user permissions can be given to existing users without needing to re-create users every single time. This usually involves integration with Keycloak, Okta, or Auth0. A good example of such an integrated data stack is HelloData, but there are more—see declarative data stacks.

But why would you invest all this energy and effort to have something run on Kubernetes? Besides the declarative approach mentioned, which is more robust than imperative approaches that tend to break down more often, especially for large projects, Kubernetes has significant advantages. The DevOps-style deployment fosters a culture of collaboration and shared responsibility through configuration YAML files checked into a git repo, which is pivotal for how data teams can work with an efficient workflow and increase productivity.

This way of working is called Infrastructure as Code, or GitOps, and is strongly related to DataOps. So, what are the toolkits for DevOps, you might ask?

Container & Orchestration:

• Kubernetes (k8s): De facto standard for container orchestration that provides scalable, cloud-agnostic deployment with declarative infrastructure management.
  • Red Hat OpenShift: Enterprise Kubernetes platform with integrated developer tools, security features, and multi-cloud capabilities.
  • kubectl: Command-line tool for managing Kubernetes clusters and debugging containerized data pipelines
  • Kustomize: Configuration management tool for Kubernetes that allows environment-specific customizations without template complexity
• Helm: Package manager for Kubernetes that simplifies the deployment of complex data stack applications with reusable charts
• Docker: A containerization platform that ensures consistent environments across development, testing, and production for data engineering workloads

Infrastructure as Code (IaC):

• Terraform: A multi-cloud infrastructure provisioning tool that enables versioned, reproducible cloud resource management for data platforms
• Pulumi: Modern IaC platform supporting multiple programming languages for infrastructure definition with strong typing and testing capabilities
• Ansible: A configuration management and automation tool that handles server provisioning, application deployment, and system administration tasks
• Koreo: A new approach to Kubernetes configuration management and resource orchestration, empowering developers through programmable workflows and structured data

GitOps & CD Tools:

• ArgoCD: Declarative GitOps continuous delivery tool for Kubernetes that automatically syncs cluster state with Git repositories
• Flux: GitOps toolkit for keeping Kubernetes clusters synchronized with Git repository configurations using pull-based deployment
• Octopus Deploy: Advanced deployment automation platform for complex multi-environment releases with approval workflows

CI/CD Platforms:

• GitHub Actions: Native GitHub CI/CD platform with an extensive marketplace ecosystem for automated testing and deployment workflows
• GitLab CI/CD: Integrated DevOps platform providing end-to-end automation from code to deployment with built-in security scanning
• Jenkins: Open-source automation server with controller/agent architecture ideal for complex, customizable build and deployment pipelines
• CircleCI: Cloud-native CI/CD platform known for fast build times and Docker-first approach to testing data engineering workflows
• Bamboo: Atlassian’s CI/CD tool with tight integration to Jira and Bitbucket for teams already using the Atlassian ecosystem

Security & Secrets Management:

• SOPS: Encrypted secrets management tool that works with PGP/age keys to secure sensitive configuration data in Git repositories
• HashiCorp Vault: A dynamic secrets management system for secure storage and access to tokens, passwords, and certificates

There’s a lot more, but these are some of the first tools you will encounter if you start scaling out your data platform on Kubernetes and use modern DevOps practices to build a data engineering platform that is maintainable and scalable, ensuring reproducible deployments and efficient collaboration across development and operations teams.

DevOps Abstraction Levels

What are the alternatives to DevOps?

DevOps isn’t binary; it’s about selecting the appropriate level of control and abstraction for your specific needs. You’re still practicing DevOps, whether you’re managing Kubernetes clusters or deploying serverless functions; you’re just operating at different levels of abstraction.

Serverless and Managed Services represent the highest abstraction level, where you focus purely on your data logic while the platform handles infrastructure concerns. Tools like AWS Lambda, Google Cloud Functions, and managed data warehouses let you deploy code and query data without worrying about servers, scaling, or maintenance. Your application remains portable, with core business logic that can typically be moved between providers, but you trade some customization for operational simplicity.

Container-as-a-Service (CaaS) platforms, such as Google Cloud Run, AWS Fargate, or Azure Container Instances, offer a middle ground. You containerize your applications (maintaining portability) but delegate orchestration complexity to the platform. You still get the benefits of DevOps practices—version control, automated deployments, Infrastructure as Code—without managing the underlying infrastructure.

Managed Kubernetes services, such as Google GKE, Azure AKS, and AWS EKS, provide another abstraction layer, offering full Kubernetes capabilities without requiring control plane management. This bridges the gap between complete infrastructure control and operational simplicity.

The key is matching your abstraction level to your team’s expertise and requirements. Start with higher abstraction levels for faster delivery, then move toward more control only when specific customizations become necessary.

Data Quality and Observability

As the data platform becomes more complex and features additional tools, it becomes increasingly sensible to have a data quality or observability stack—tools to have an automated overview of the health of your data platform.

Below are some of the standard tools (without getting too lengthy) that we haven’t covered and were not mentioned in Part 1:

• ELK Stack: Elasticsearch, Kibana, and Logstash. Reliably and securely take data from any source, in any format, then search, analyze, and visualize.
• Prometheus: Open-source monitoring system and time series database.
• DataDog + Metaplane: Monitoring and security platform for developers, IT operations teams, and business users in the cloud age. DataDog recently acquired Metaplane, an end-to-end data observability platform that catches silent data quality issues before they impact your business.
• Datafold: Comprehensive data monitoring to prevent downtime and detect data quality issues early.
• Soda: Soda is a data quality testing solution, with parts of it open-source, like data quality testing for the modern data stack (SQL, Spark, and Pandas).
• Monte Carlo: Enterprise-ready with extensive data lake integrations
• Bigeye: ML-driven automatic threshold tests and alerts

AI-Enhanced Workflow Development

New AI-enhanced tools with LLMs or MCPs are being invented and are already useful today.

For example, for data engineers, there are dedicated IDEs or integrations into MCPs—especially agentic workflows:

• nao: An AI-enhanced editor specifically for data engineers. In its early days, it understands dbt and can create and run pipelines.
• MCP server for DuckDB and MotherDuck: Makes your editor autonomously query the underlying database on the fly.
• Claude Code: An agentic coding tool that lives in your terminal, understands your codebase, and helps you code faster through natural language commands.
• dbt MCP: A MCP server provides tools to interact with dbt autonomously, like running dbt build or docs, etc.
• Rill MCP Server: Exposes Rill’s most essential APIs to LLMs. It is currently designed primarily for data analysts.

Also, check out Faster Data Pipeline Development with MCP and DuckDB, which explains MCP in more detail and directly showcases some of the use cases.

Soft Skill: Communication, Business Requirements

As AI workflows reduce the need for coding, business acumen and soft skills become even more crucial. This section focuses on the human aspect of communication within the organization or among team members, and gathering the right business requirements before developing a platform or solution that may not be needed in the first place.

Essential Soft Skills

Business understanding is crucial for practical data engineering. This means being genuinely interested in business nuances, actively listening to domain experts, and developing strong communication skills for requirements engineering, which significantly overlaps with traditional BI engineering roles.

Cross-functional collaboration is equally important. Data engineers must translate technical constraints and possibilities into business terms for stakeholders, while also understanding their pain points and priorities. This includes stakeholder management, documentation skills, and the ability to ask the right questions to uncover hidden requirements and assumptions.

While you can be a technical expert without these skills, combining technical expertise with strong business understanding and communication will set you apart. It helps you solve real business problems and deliver measurable value—something we should always keep in mind.

Building Your Data Engineering Toolkit

Wrapping up these two articles on the in-depth toolkit for data engineers, I hope you’ve learned a tool or two that will improve your workflow as a data engineer or in the data field.

Hopefully, you won’t be overwhelmed by all the links. Again, it’s not meant to be a toolkit for everyone, but instead provides pointers for the direction you’d like to explore when starting or when you want to venture into a slightly different area of data engineering.

We’ve gone from fundamental to advanced DevOps skills and learned along the way:

• Developer tools and programming languages in Part 1 and the sophisticated ecosystem of modern data engineering in Part 2.
• SQL databases and Python as your foundational toolkit, with analytics and BI platforms for presenting insights.
• DevOps and Infrastructure as Code for scalable deployments with Kubernetes.
• Data quality and observability solutions for maintaining platform health.
• Emerging AI-enhanced workflows that are reshaping how we build data pipelines.
• Technical expertise alone isn’t always enough; strong communication skills and business understanding transform data engineers into 10x contributors, delivering real value to the business.

If you want to learn more tips and tricks about the toolset, please follow the MotherDuck newsletter for the latest news about DuckDB, which usually contains great insights and tools for working with data through DuckDB or MotherDuck. You can also try MotherDuck, which allows you to handle many data use cases in a notebook environment with many of the tools mentioned in these articles.

If you have a toolkit you use every day as a data engineer or a unique tool that cannot be found in the two parts, please let me know on social media in the comments. I’d be happy to know what you use as your core toolkit for everyday work.

Full article published at MotherDuck.com - written as part of my services

This article focuses on the building blocks of data engineering work, such as operating systems, development environments, and essential tools. We’ll start from the ground up—exploring crucial Linux commands, containerization with Docker, and the development environments that make modern data engineering possible. We look at current programming languages and how they influence our work—providing a comprehensive overview of the tools of a modern data engineer.

Before we start, you don’t need to know everything discussed here, but over time, you may use all of them in various roles as a data engineer at different companies. I hope this article will give you a good overview and guidelines on what is essential and what is not.

Again, each selection might differ slightly depending on the company’s setup, preferred vendors, and whether it uses a low-code or a building approach. Let’s start with the first choice you must make at any company, the operation system to work on.

Operating Systems & Environment

Before starting as a data engineer, your laptop, operating system (OS), and environment are your first choices. Here, we discuss the different OSs and virtualization you will encounter, such as Docker and ENV variables, to configure different environments.

Operating System Choices (Windows/Mac/Linux)

Choosing the right operating system might seem significant. Primarily, it’s a preference for what you like and know. Still, there is the fact that most data platforms that run on a server will run on a Linux-based OS system. Working on Linux OS on the client might give you skills you can reuse, but you can also have that with Windows with WSL¹ and MacOS running a Darwin-based Linux.

Your employer also defines it. If you are a Microsoft shop, you use tools such as Power BI, Visual Studio (not Visual Studio Code), and C#. This requires using Windows or at least a VM with Windows.

If you work at a startup and need great hardware that is easy to use, the company will probably provide you with the latest MacBook with MacOS installed. However, if you are a power user or need your Dotfiles, you may not use anything other than a Linux-based operating system. We will look later at fundamental Linux commands that make the life of every data engineer easier.

Virtual Machine (VM)

As mentioned, you could run MacOS and Windows in a VM with VMware or Parallels. These are not native installations, but close to it, and they allow you to do most things.

The same goes if you are on Windows; instead of using WSL, which sometimes can get tricky with companies’ proxies and network routing, you could use a Linux VM locally or somewhere hosted that you just SSH into or an advanced example with Nix. There are other solutions to explore; e.g., your whole machine could be a VM provided by your company or deploy a VS Code server to run VS Code instances inside your company network.

ENV variables

The next layer that you commonly use is ENV variables. This is already a little more advanced. But think of your reproducible environments with your co-workers or managing different environments (dev/staging/prod) instead of hard copying all settings, which won’t work on other environments with different OS or other expectations.

If you type env in a Linux-based OS terminal, you can see all your local env sets. To illustrate some, I have set these ENVs:

1
2
3
4
5
6
7

❯ env
AIRFLOW_HOME=~/.airflow
SPARK_HOME=~/Documents/spark/spark-3.5.1-bin-hadoop3.3
MINIO_ENDPOINT=http://127.0.0.1:9000
GITHUB_USER=sspaeti
AWS_SECRET_ACCESS_KEY=my-secure-key
AWS_ACCESS_KEY_ID=my-access-key

These can be set in a projects-repositories folder, usually in .env, and which will be picked up automatically. However, the recommended approach is using SSO CLI tools (like aws sso login or gcloud auth login), which will automatically populate credentials in the expected locations, or alternatively adding them to your shell config (~/.bashrc, ~/.zshrc).

Docker and Container Images

Another virtualized environment is Docker, and specifically Dockerfiles. Docker is the engine that runs your Dockerfile on all platforms and architectures, letting you create a container image and build it for Linux on a Windows machine.

That makes containers so powerful: you can package and containerize complex data engineering requirements into a single Dockerfile, and everyone can run it on any machine—whether locally, in CI/CD pipelines, or orchestrated in Kubernetes clusters. Think of container packages on ships that transport goods; the breakthrough was the standardized container size that fits on every boat; every harbor could maneuver them. Similarly, container images have become the standard for packaging data and software ecosystems, with formats originally defined by Docker now being widely supported across different container runtimes and platforms.

A simple nginx (webserver) example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# Use the official NGINX image from Docker Hub
FROM nginx:latest

# Copy your custom NGINX configuration file (if you have one)
COPY nginx.conf /etc/nginx/nginx.conf

# Copy static website files to the appropriate directory
COPY . /usr/share/nginx/html

# Expose the port NGINX listens on
EXPOSE 80

Docker also supports different instructions that you can use in a Dockerfile.

Linux DE Fundamentals

Even though you might use Windows, Linux is key to a data engineer. You don’t need to be an expert, but you shouldn’t be afraid of command line tools and know some basic Linux commands. And be aware that some of them are powerful.

Opening and Editing a File with Nano/Vim

Editing or creating a new file might not be as easy as it seems. Command line text editors such as Nano or Vim can be used for this task. Recommended is Nano, which displays the shortcuts to save or exit. Vim can be intimidating at first, but it’s a worthwhile investment when working 8 hours a day on the terminal, even more so Vim Motions.

Basic Linux Tools and Commands

In addition to the Linux basic commands you have probably used or encountered like cp, mv, ssh as seen below, which are also super helpful on a server, we focus on the data engineering Linux commands you run on your laptop, where you can install things.

Most tools are Python-related to achieve the core tasks of a data engineer: ingestion of data, transforming and serving it to the organization or users. But the additional DE Linux commands I often use to quickly check an API, copy something over, or check processes are:

• curl: Quickly check an API is available through the cmd line.
• make / cron: Simple orchestration with the command line. More on this in the next chapter
• ssh / rsync: Ssh to connect to another machine and Rsync for a fast, versatile, synchronization tool to quickly back up or move data from your machine to the server.
• bat: Show data of a file nicely format and git integration.
• tail: Displays the last part of a file, which is helpful if the file is big and cat/bat would take long.
• which: Locate a program in the user’s path to check if the right tool is running.
• brew: MacOS-specific package manager is the easiest way to install tools and cmd line utils.

Related to the above basic Linux commands:

• grep: Used for everything attached to an existing run. E.g. quickly search AWS env variables:

1
2
3
4

❯ env | grep AWS
AWS_ACCESS_KEY_ID=my-access-key
AWS_BUCKET=my-bucket
AWS_SECRET_ACCESS_KEY=my-secret

• ps aux and htop: To check the current process. Ps is also handy in combination with grep (ps aux | my-program.py)
• rg and fzf: Ripgrep (rg) is a recursive line-oriented search tool that searches through all files, and fzf is a fuzzy finder. In combination, you can interactively search fuzzy find the content of Python files in the current folder easily with rg -t python "def main" . | fzf. (Also check out Recursive Search in Terminal with fzf, this will change your cmd-line life with reverse search ctrl+r).

Simple Orchestration

The core responsibility of a data engineer is to orchestrate different jobs in the correct order and fully automate them. We use data orchestrators (Airflow, Dagster, Prefect etc.), but Linux also covers us.

Makefile and cron jobs are out of the box and installed on every Linux system. For example, Makefiles let us store a combination of commands like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

API_URL := "https://api.coincap.io/v2/assets"
DATA_DIR := /tmp/data

etl: extract transform load

extract:
  mkdir -p $(DATA_DIR)
  curl -s $(API_URL) | \
    jq -r '.data[] | [.symbol, .priceUsd, .marketCapUsd] | @csv' > \
    $(DATA_DIR)/crypto_raw.csv

transform:
  ./scripts/transform_data.sh
    
load:
  cat $(DATA_DIR)/crypto_raw.csv | \
    sort -t',' -k3,3nr | \
    head -n 10 > $(DATA_DIR)/top_10_crypto.csv

clean:
  rm -rf $(DATA_DIR)/*

Running make extract will create download data from the HTTPS API and store it as CSV, which we can check with tail:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

❯ make extract
mkdir -p /tmp/data
curl -s "https://api.coincap.io/v2/assets" | \
                jq -r '.data[] | [.symbol, .priceUsd, .marketCapUsd] | @csv' > \
                /tmp/data/crypto_raw.csv

❯ tail -n 3 /tmp/data/crypto_raw.csv
"ZEN","25.2499663234287359","399199442.5767759717054100"
"SUSHI","1.4507020739095067","381986878.5063751499688694"
"JST","0.0384023939139102","380183699.7477109800000000"

Combining these commands can be quick and super powerful. Make is just one example of storing and checking the commands into git so everyone can use them.

Crontabs are another way to schedule them daily, for example.

Pipeline command: Join different commands together |

In line with the Unix Philosophy, to make one tool do one thing as best as possible, you can combine “pipe” different tools with | as we’ve seen examples already above with grep and others.

Here is another example of checking if any Python packages for SQL have been installed

1

pip freeze | grep SQL

This allows the making of data pipelines within the terminal and a single cmd line by stacking different operations together. Example of powerful command chaining with pipes:

1

❯ bat /tmp/data/crypto_raw.csv | tr -d '"' | cut -d',' -f1,3 | sort -t',' -k2 -nr | head -n 4 BTC,1920648934960.3101078883559601 ETH,386675369242.2018025632681003 XRP,161734797349.4803555794799785 USDT,137222181131.1690655355161784

The pipeline reads the above CSV file and extracts the coin name and market cap only (using cut), removes the quotes (tr), and then sorts by the market cap value numerically in descending order to show the top 4 biggest cryptocurrencies by market capitalization.

Data Processing

Another example could be data processing within the command line—e.g., quickly splitting a large CSV that you are unable to open with a text editor:

1
2
3
4
5

# Split large CSV while keeping header
head -n1 large_file.csv > header.csv
split -l 1000000 --filter='tail -n +2' large_file.csv chunk_
# Add header back to each chunk
for f in chunk_*; do cat header.csv "$f" > "with_header_$f"; done

I hope you can imagine how you could build any small, efficient data pipeline with a Makefile and the Pipe commands.

Developer Productivity

Next, we will look at the newer tools that can be added above the terminal and CLIs: powerful IDEs, notebooks, or workspaces, and git for version controlling everything.

IDE (Working environment)

An integrated development environment (IDE) is where we program our code and get code completion, linters, and AI assistance to make us (hopefully) more productive.

Popular IDEs are with their used based on the StackOverflow Survey 2024:

• Visual Studio Code (73.6%) - Microsoft’s lightweight but powerful source code editor with extensive plugin support and language coverage.
• Visual Studio (29.3%) - Microsoft’s full-featured IDE, powerful for .NET development and enterprise applications.
• Other editors sorted percentage-wise are IntelliJ IDEA (26.8%), Notepad++ (23.9%), Vim (21.6%), PyCharm (15.1%), Jupyter (12.8%), Neovim (12.5%), Sublime Text (10.9%), Eclipse (9.4%), Xcode (9.3%)

Not even on the map 2024 were the IDEs that go all in with AI:

• Cursor - A VS Code-based editor explicitly built for AI-assisted development, featuring GitHub Copilot integration and specialized AI tooling for code completion and refactoring.
• Windsurf - An AI-first code editor designed to streamline development workflow with features like natural language code generation and intelligent code suggestions.
• Zed - A high-performance, multiplayer code editor with AI capabilities created by former Atom developers.

Codespaces and Workspaces

In addition to IDEs that are usually installed locally, we also have codespaces (or workspaces, depending on the naming) that live in the browser. These are super handy because everyone has the same environment, and the days of “does not work on my machine” are gone.

These tools include GitHub Codespaces, Devpod, Replit, Stackblitz, CodeSandbox Gitpod, and many others.

Notebooks

In addition to IDEs and Codespaces, you can use a notebook that runs locally or in the cloud. This option is generally more flexible and allows you to visualize results and document the code. However, putting it in production has a downside: It’s harder to restart, backfill, or configure with different variables.

It’s more flexible and easier to get started, but transitioning notebooks to production remains challenging even on platforms like Databricks, which are designed to support a development-to-production workflow.

Notebooks like Jupyter Notebook / JupyterHub, Apache Zeppelin, or Databricks Notebook. Newer versions of Jupyter Notebooks with more integrated features and a robust cloud behind them are Deepnote, Hex, and Count.co, Enso, or MotherDuck, which combines the flexibility of notebooks with the power of DuckDB’s analytics engine.

Git Version Control

Git is probably the most used version control in data engineering nowadays. There was a time of TortoiseSVN and others.

As a data engineer, you need to version your code and product to easily roll back in case of error or work together as a team. The most common git workflow are:

1
2
3
4
5
6

git pull origin main # Pull latest changes
git status # Check status of your changes
git add pipeline.py #stage
git commit -m "fix: update extraction logic for new API version" #commit
git push origin main # Push to remote repository
git checkout -b feature/new-data-source # Create and switch to a new branch

For more complex operations, consider using a Git GUI client. Some popular options include GitKraken, SourceTree, Lazygit (terminal UI), and many more.

Data Engineer Programming Languages

Before we wrap up, let’s look at a data engineer’s programming language. This will change depending on whether you are working more on infrastructure, pipeline, or business extraction.

The most prominent language you will use is still SQL, as the language to query each BI tool, doing most transformations with dbt and others, and even having an API on the most popular DE libraries makes it the best first language to master. Just after, especially if you build a lot of data pipelines and do a bit above basic transformations, you won’t get around Python. Python is the tooling language of a data engineer; think of it as the Swiss army knife.

Lastly, if you are in infrastructure and need to deploy the data stack, you primarily work with YAML as a definition language for Helm, Kubernetes, Terraform, or other deployments. You could write some Rust if you are developing infrastructure and performance-heavy optimization.

We can see the most popular languages as with the StackOverflow 2024 data, query with DuckDB with a shared DB on MotherDuck—simply sign up (if you haven’t) and create a token to query the database with this SQL-query:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

┌─────────────────────────┬───────┬──────────────────────────────────────────┐
│        language         │ count │                  chart                   │
│         varchar         │ int64 │                 varchar                  │
├─────────────────────────┼───────┼──────────────────────────────────────────┤
│ JavaScript              │ 37492 │ ████████████████████████████████████████ │
│ HTML/CSS                │ 31816 │ █████████████████████████████████▉       │
│ Python                  │ 30719 │ ████████████████████████████████▊        │
│ SQL                     │ 30682 │ ████████████████████████████████▋        │
│ TypeScript              │ 23150 │ ████████████████████████▋                │
│ Bash/Shell (all shells) │ 20412 │ █████████████████████▊                   │
│ Java                    │ 18239 │ ███████████████████▍                     │
│ C#                      │ 16318 │ █████████████████▍                       │
│ C++                     │ 13827 │ ██████████████▊                          │
│ C                       │ 12184 │ ████████████▉                            │
├─────────────────────────┴───────┴──────────────────────────────────────────┤
│ 10 rows                                                          3 columns │
└────────────────────────────────────────────────────────────────────────────┘

Beyond Languages

Beyond programming languages, you must get to know various databases and their concepts, such as relational database theory. It does not matter which SQL dialect you learn, as they are all related, but knowing the fundamentals of a specific database, such as Postgres, DuckDB, or a NoSQL database, will help you on your journey.

Python libraries and frameworks are the last we observe and where you can spend most of your time. Instead of learning as many as possible, I suggest investing in a few used at your company and where you benefit most.

Typical starter libraries include DuckDB (a powerful in-memory transformation library and database with scale-up capabilities via MotherDuck²), Pandas (flexible data manipulation), PyArrow (optimized for columnar data), Polars (fast and scalable DataFrame library), and PySpark (for distributed data processing with Apache Spark).

Python Libraries

There are many more libraries available, especially when you need to quickly access an API or perform a task that a CLI can’t. Some key libraries can be beneficial depending on the use case you are working on.

Data Ingestion:

• Requests - HTTP library for API queries and web scraping
• BeautifulSoup - HTML parsing library for web scraping

Developer Tools:

• uv / pip - Package installers for Python, with uv being a modern, fast alternative to pip
• Ruff - Fast linter and code formatter
• Pytest - A testing framework for Python

Data Validation:

• Pydantic - Data validation for Python objects
• Pandera - Schema validation for dataframes
• Great Expectations / OpenLineage - Data quality validation framework and data lineage tracking tools

We could go on forever. Libraries exist for virtually everything: data ingestion, orchestration, BI tools, you name it. We could discuss setting up a Python project (it’s not a solved problem, and there are many ways of doing it), discuss DevOps and how to use a simple Helm script, set up a local storage system that mimics S3, and more.

Wrapping Up

Instead, we wrap it up, and I hope you enjoyed this article. It gave you an overview and a sense of how much is asked from a data engineer these days. But as this might be overwhelming, I suggest always focusing on fundamentals and, second, taking it step by step. It’s better to understand why than skip over it quickly. — Also, as we are in the AI area, use ChatGPT to explain a command or a CLI tool to you; it will do a much better job than any Google Search.

We’ve covered the foundational tools and environments of modern data engineering, skills that are often overlooked but crucial for any data engineer. From selecting the proper OS and virtualization setup to mastering Linux fundamentals and CLIs, these building blocks enable efficient data pipeline development without always requiring complex tools.

This foundation reminds us that sometimes the simplest solution is the most effective—a well-chosen Linux command can often replace a complex toolchain. I hope that these technical skills, provided by a modern data engineer, will help you along your journey when working from the command line on your machine.

MotherDuck strives for modern data development and developer productivity. For instance, its approach to developer productivity allows seamless scaling from local development to production: developers can work with DuckDB locally using path: "local.duckdb" for their development environment, then simply point their production environment to MotherDuck with path: "md:prod_database". This lets engineers focus on feature implementation while MotherDuck handles the scaling and performance.

For a practical example, check out this implementation in the Deep Dive - Shifting Left and Moving Forward with MotherDuck: 

Full article published at MotherDuck.com - written as part of my services

At its core, GenBI delivers an unreasonably effective human interface, where we iterate quickly, based on BI-as-Code. A simplified version looks like this:

But what makes this possible? The key lies in the declarative BI stack as discussed in Part 1 - where dashboards and metrics are defined as code (like covid_dashboard.yaml) rather than hidden behind graphical user interfaces. The declarative approach gives AI models the context they can understand and work with: structured definitions of business metrics, relationships between facts and dimensions, and visualizations.

In this article, we want to explore the possibilities of GenBI today.

Understanding GenBI

Generative business intelligence changes how people interact with data, enabling AI-driven analytics. By combining the power of Generative AI, GenBI makes analytics more accessible through new human interaction methods. Generative BI enables these use cases, from creating dashboards to natural language querying (typing or talking), all the way down to the data model generation if you only have the source tables. We explain the model from a top-down, conceptual idea of what we need, and we generate dashboards, metrics, data models (snowflake/star schema, relationships, joins, even grains), entities, or even data warehouse architectures.

A key aspect of GenBI is its declarative nature, as we discussed in the Declarative Data Stack. Whether the entire stack or not, one of the most critical layers is the Metrics Layer. Generative AI needs context to understand the data model and the company’s business. The metrics layer and its semantic (also called semantic layer) are relevant for this process, as the metrics hold the semantic understanding. Let’s see GenBI in action based on the declarative dashboard:

One step further: Data modeling languages such as LookML, MDX, MAQL, Malloy language, or SQL are critical. These languages allow humans to describe and model our metrics and KPIs, from which AI models can learn. They are expressive and declarative and define the company’s business logic.

To illustrate how GenBI leverages a declarative dashboard, let’s see how Al can automatically add a new measure for average fare cost per mile to our existing metrics within the text editor:

In summary, combining AI models with BI tools allows business users to query data, generate reports, and derive insights using conversational language, making data analytics more accessible.

Evolution from Traditional BI to GenBI

Before we discuss GenAI, let’s understand the difference between today’s BI and GenBI.

To grasp that, let’s quickly follow the evolution of SQL, and how it shaped the BI tools. From traditional data marts and materialized views queried directly by the BI tool, we’ve built data warehouses, lakes, or lakehouses with tools such as dbt and data warehouse automation tools. Sometimes, we added an OLAP cube if we needed fast response times, typically modeled as OBT tables or wide denormalized tables.

In all of the different ways, SQL has been at the heart, even more with complex data pipelines and semantic layers. Even Large Language Models (LLMs) made it into SQL statements.

With GenBI, this will not change as the BI tools need to execute SQL at the end of the day. But what is important is that the metrics and the SQL statement are declarative stored. Therefore, it might be easier to work with YAML to maintain complex definitions, at least until we extend the SQL syntax for analytics.

How GenAI Powers BI and compare to GenBI

The main difference between traditional BI and GenBI is understanding the semantics behind SQL with AI. What does that mean?

SQL is a declarative language; therefore, the AI model can learn from the queries. If we feed it with more data, such as the metrics in a metrics layer as part of the BI tool, and if available, the data model (DDL), the model will have a semantic understanding of queries. If you have a declarative dashboard tool such as Rill, you can train the model on the dashboards so it learns how we create them within the company, too.

GenAI has an LLM trained on our business intelligence artifacts. In return, it can generate visualizations in the form of dashboards and metrics in the form of SQL aggregation for us.

The other part of GenBI is what we use every day. With the rise of ChatGPT and similar tools, we can interface with our tools in a more human-like manner. Instead of writing SQL, we can speak or write natural language to query data or present it in a dashboard, compared to hand-crafting. These are two use cases, but the applications are endless, and new ways of interaction arise every day.

Ultimately, if we talk about GenBI, we must mention GenAI as the driving force and integrate AI technologies with BI tools and data sources. GenAI is a broad term encompassing all generative AI technologies, while GenBI is a specialized application of GenAI focused on business intelligence.

BI-as-Code: The Foundation of GenBI

We’ve gone through a long evolution of mouse-clicking first dashboard tools, where everything you define lives within the UI. These tools produce unwieldy exports with hard-coded IDs and visual coordinates, often spanning thousands of lines of XML/JSON that are hard to version or modify systematically.

The benefits of each are clearly visible:

1. Graphical and traditional BI approach: Initially fast but slow with iterations. With manual changes, error prone.
2. Code-First Approach: Scales well with complexity. Works well with teams.
3. GenBI: Instant interactions through general knowledge LLMs. The best of both worlds makes it more approachable for businesses and users with the natural interface.

Benefits of Code-First Analytics

Today, newer code-first approaches let you define dashboards declaratively, bringing all the advantages of the Declarative Data Stack: automation, versioning, and separation of business logic from implementation. This approach offers the best of both worlds – an intuitive UI for design while producing clean YAML definitions that can be versioned and bulk-modified with search + replace across all dashboards for example.

Code-first enables a smaller data team to do more. Think of the Ruby on Rails developer, who does end-to-end from changing the database theme to the front of the app. Instead of multiple people understanding every little column of a Shopify database model, we can prompt the BI model that has been trained on the physical data tables and potentially 100 or reference implementations from GitHub to generate a “dashboard for the sum of orders per month”. This will not be 100% accurate, but you might get 80% of the work within a very short time.

Instead of generating just the dashboard, it could create a central metrics layer repository to enhance versioning and maintainability, which would improve governance.

But beyond maintainability - it creates a semantic foundation that AI can understand. When dashboards are defined in code, they explicitly declare:

• The metrics being visualized and how they’re calculated
• The relationships between different data dimensions
• The business logic behind aggregations and transformations
• The visual hierarchy and organization of information

While traditional BI tools usually hide these relationships in opaque UI configurations, code-based definitions make them machine-readable and learnable. The metrics layer becomes a natural extension, where business definitions are codified consistently and versionable, creating the foundation for an AI model to understand the business context. This enables automation and the generation of visualization.

This is why declarative is a prerequisite to GenBI. While AI can control visual interfaces, this approach is inefficient for complex business logic. The data model joins, metric aggregations and bus matrix require explicit declarations that AI can parse and understand; print-screened images are insufficient.

How Would a GenBI Workflow Look Like?

BI-as-code enables GenBI workflows. A potential workflow would involve a GitHub PR based on a GitHub repo as the persistence layer, where humans and AI brainstorm.

Considering the initial flow chart, where humans could create the prompt within a PR, the model generates its artifacts and commits to it. The human analyzes, verifies, and iterates on the prompt until he is happy. When finished, he approves and merges the PR, which will be deployed to production.

This would allow for an excellent review stage, during which humans and AI can iterate with a persistent store on GitHub. This could work for visualizations (dashboards), business logic (metrics layer), and data models (DDL for data warehouses).

From Conceptual to Physical Data Model

If we zoom out, we can use GenBI to model the conceptual all down to the physical layer in a top-down approach.

Mapping each stage to a BI artifact, with GenBI’s strength being more to the left, simply because we will have more context:

• Conceptual = Dashboard generation
• Logical = Metrics layer modeling
• Physical = Table and Joins (FKs) modeling

Core Components and Architecture

If we look at the landscape of GenBI, it’s not yet well defined. Below is an attempt to highlight the different components that interact with each other in greater detail to achieve GenBI.

The most critical core components of GenBI are:

• Business Intelligence: BI-as-Code tool at the center and interface for human interactions. Declarative dashboards are stored with BI-as-Code and integrated with the metrics layer. The metrics layer serves as a repository of business definitions, including measures and data model relationships (joins, star/snowflake schema). Together with the declarative dashboards and metrics, they provide essential semantic context back to the AI engine for generating and modifying BI artifacts.
• GenBI Core: Consists of three key components working together:
  • Natural Language Interface: Enables human-like interaction with the system
  • AI Engine: Processes queries and understands the business context
  • BI-as-Code Generator: Transforms AI understanding into concrete, declarative code artifacts like YAML dashboard definitions and metrics configurations
• External Knowledge: Combines general knowledge from external LLMs with business-specific context through RAG (Retrieval-Augmented Generation). While LLMs provide a broad understanding, the RAG store enriches this with internal documents, legacy code, and company-specific data models.
• Data Sources: Serve as the foundation where the BI-as-Code Generator queries to validate and execute the generated artifacts against actual data. Represents any kind of database or files.

Key Components for Successful GenBI

To make BI successful, it needs to provide an instant overview of company performance to allow the leadership to make critical decisions fast. BI should boost data efficiency by automating repetitive tasks across the organization, such as updating daily active users and sending them to relevant stakeholders.

To achieve successful GenBI, we need to highlight at least two main components that need to be well integrated into GenAI to make BI successful:

• Dashboards: Quickly visualize the hard work of data engineers, the ingestion, wrangling, cleaning, and transforming of the data pipeline for the business and users.
• Metrics Layer: Auto-generate definitions for standard metrics like monthly active users, year-to-date revenue, etc.

If we can further automate this process, we will overcome one of the biggest problems of BI: the bottleneck of the data or BI teams to ship or publish data artifacts. Also, the BI stack usually requires skills different from those of a domain expert. We have won a lot if it empowers domain experts who understand the business, how it runs, and how to present the numbers to tell a story.

You need to understand the company and its operations and know how to present the numbers in a way that tells a story.

This is usually where technical data professionals care least, so a trained model that helps us would significantly boost productivity. This is precisely where GenBI helps address these challenges.

Presentation Layer: Dashboards

Dashboards in GenBI serve as the primary interface between business insights and decision-makers. The iterative nature of dashboard development makes them particularly well-suited for AI assistance, with two distinct phases:

The initial creation process focuses on rapidly transforming business requirements into visualizations. GenBI excels here by understanding the context from the metrics layer and suggesting appropriate visualizations based on data characteristics and best practices.

GenBI truly shines in the iterative process of incorporating new features or data into the dashboard. It can provide feedback quickly, suggest improvements, and adapt visualizations based on user needs and data patterns. As dashboards are never truly finished, GenBI is an invaluable partner in continuous refinement, helping maintain consistency while reducing technical overhead.

The Power of the Metrics Layer

The metrics layer serves as GenBI’s semantic foundation, providing a declarative way to define and maintain core business definitions, including measures and data model relationships. GenBI creates a single source of truth that both humans and AI models can understand by centralizing these definitions in code. This is essential for consistently interpreting business logic across the organization, from simple metrics to complex star/snowflake schema relationships.

Beyond basic measure definitions, the metrics layer captures the semantic relationships between business concepts through explicit data model definitions like joins and fact/dimension relationships. This structured context enables the AI engine to understand individual metrics and how they relate to each other and the underlying data model. This comprehensive semantic understanding creates a powerful foundation for the AI engine when combined with declarative dashboards from the BI-as-Code tool.

The engine can generate contextually appropriate visualizations, suggest relevant metrics for specific business questions, and understand complex business calculations and their relationships. It can even auto-generate new metric definitions based on existing patterns while validating generated artifacts against the actual data model, ensuring consistency and accuracy in the business intelligence layer.

GenBI in Action

Let’s examine GenBI’s practical implications through example prompts. Later, we will look at an actual implementation with Rill.

Prompts for Dashboard Creation

Let’s start with the user perspective and the prompts. Below are typical natural language requests (whether written or spoken) on how to “prompt engineer” a beautiful, fully functional dashboard. These prompts range from simple to advanced to interactive use cases that GenBI can help us with.

From simple metrics and visualizations:

Create a metric for revenue based on data in my ORDERS table.

Create a chart showing order revenue broken down by product category in the last year.

Create a visualization of revenue to sales goal this month, and set the sales goal at $28,283.

To more formatting and style update refinements:

Optimize for a cleaner visual aesthetic based on Edward Tufte's design guidelines.

Update on-brand color palette for this visualization based on the colors from our logo (upload logo)

Create more advanced dashboards with a specific focus:

Create a cohort analysis dashboard showing customer retention rates over 12 months, with the ability to filter by acquisition channel and highlight cohorts that exceeded 80% retention

Build a drill-down capable sales dashboard that starts with global performance but allows users to click through to regional, store, and individual product level metrics, maintaining consistent visual language throughout the hierarchy

Moving beyond basic visualizations, you can iteratively refine dashboards:

1. Add a detailed table below the main "trend chart" showing monthly breakdowns
2. Update the color scheme to use black for current year data and gray for previous year comparisons
3. Add sparklines to the table columns showing 6-month trends
4. Enable drill-down on product categories to show individual SKU performance
5. Add conditional formatting to highlight values that are >10% below target in red
6. Create a collapsible section for additional metrics like margin and inventory turnover
7. Add hover tooltips showing YoY growth percentages

These are examples of prompt dashboards. Let’s look at metric-specific generations.

Prompts for Metrics Generation

Exclude all refunds and returns from my revenue metric.

From basic metric definitions:

Create a metric for total revenue as the sum of order prices, define monthly active users based on our users table and calculate the average order value from our orders table

Generate star-schema data model metrics based on source DDL:

Analyze my source tables DDLs from Stripe (`payments, subscriptions, customers`), SAP (`orders, inventory, suppliers`), and our CRM (`interactions, support_tickets`) to create a consolidated dimensional model.

Generate appropriate fact and dimension tables, define key metrics like revenue, customer lifetime value, and order frequency, and include common aggregations and filters.

Iterative refinement through conversation:

These examples showcase the range from simple aggregations to complex business logic while highlighting the conversational nature of GenBI and its ability to understand and implement business requirements iteratively.

Practical Implementation with Rill Developer

Today, Rill Developer uses GenAI to create dashboards or metrics based on your data sources or models. It also stores all your sources, models (SQL queries), and dashboards in YAML and has a live editor for visually changing them.

You can right-click on your model to use AI with the comfort of your BI tools (this works in the OSS and Cloud version):

If you click, generate dashboard, that would look like this based on the infamous NYC taxi data set. Notice the beautiful pivot table at the end:

This implementation demonstrates the future of one-click dashboards with generative AI, where LLMs like GPT can generate and modify dashboard definitions directly in code.

Rill’s BI-as-code philosophy means dashboards are defined entirely in code and that large language models, like OpenAI’s GPT series, can generate based on these definitions. Rill today does an OpenAI call to get domain-specific understanding, e.g., if the data model is a certain industry.

Incorporating code generation into development environments makes the tool much more user-friendly and can significantly improve a single BI engineer’s productivity. Rill Developer also provides software engineers with a fast feedback loop, enabling them to edit code rapidly, visualize metrics, and instantly preview dashboards before deploying them into Rill Cloud or a self-managed environment.

What else does Rill Developer bring to the table? Rill is a CLI-first BI tool installed with a single command, opening up all kinds of new use cases, like embedding a complete GenBI tool as part of your data pipeline. Compared to traditional BI tools, that won’t be possible any time soon. With that and its declarative approach, its integrations in the most common OLAP engines and common sources meet all GenBI requirements.

Get Started with Rill Developer and GenBI Today

One-line installation:

1

curl https://rill.sh | sh

To run as a sample project:

1

rill start my-rill-project

Check the Quick Start for more information.

Future of GenBI

After seeing what GenBI is, I understand the evolution from traditional BI to GenAI, using BI-as-Code as the foundation to enable powerful AI use cases that generate full dashboards and metrics based on existing data sources. We’ve seen the core components GenBI needs and how GenBI looks in action, both from the user perspective with prompts and from the implementation side with actual products that have implemented GenBI. For businesses, this means faster time-to-insight and democratized access to data analytics without sacrificing the robustness of traditional BI approaches.

So, what’s the future of GenBI? While this is difficult to predict, I’m confident Rill Developer will be at the forefront, enabling users with simple business requirements to access the entire BI stack—maybe even beyond. In the following article, we’ll look at how a declarative data stack could be implemented, integrating the BI part and end-to-end what we discussed in Part 1 and the Rise of the Declarative Data Stack.

Full article published at Rilldata.com - written as part of my services

As we see more analytics engineering and software best practices, embracing codeful, Git-based, and more CLI-based workflows, the future looks more code-first. Beyond SQL transformations, across the entire data stack. From ingestion to transformation, orchestration, and measures in dashboards—all defined declaratively.

But what does this shift towards declarative data stacks mean? How does it change how we build and manage data stacks? And what are the implications for us data professionals? Let’s find out in this article.

A Brief History of Declarative Systems

Often, we forget how hard it was in the old days.

Let’s take Fortran, one of the earliest high-level programming languages. It was revolutionary for its time, but it required programmers to think in terms of the computer’s architecture and know the ins and outs of everything.

Or SQL, did you know it’s declarative? Or Markdown, HTML, or Kubernetes? Most of them are growing in enthusiasm and support each year¹, but why? The below table gives you an idea of how declarative is more straightforward, whereas imperative is very verbose and hard to read.

import numpy as np
result = np.mean(np.array
  ([1, 2, 3, 4, 5]))

PROGRAM AVERAGE
  REAL :: numbers(5), sum, average
  DATA numbers /1.0, 2.0, 3.0, 4.0, 5.0/
  sum = 0.0
  DO i = 1, 5
    sum = sum + numbers(i)
  END DO
  average = sum / 5
  PRINT *, 'Average is:', average
END PROGRAM AVERAGE

**Bold text**

let text = "Bold text";
console.log("\x1b[1m" 
+ text + "\x1b[0m");

SELECT * FROM users 
WHERE age > 18;

for user in users:
    if user.age > 18:
        result.append(user)

kind: Deployment
metadata:
  name: my-app
spec:
  replicas: 3
  template:
    metadata:
      labels:
        app: my-app
    spec:
      containers:
      - name: my-app
        image: my-app:1.0

1. SQL for database queries
2. Functional programming languages
3. Declarative UI frameworks (e.g. Rill, Evidence, React)
4. Configuration management tools (e.g. Kubernetes)

Examples of Declarative Data Stack (Airflow/dbt/Kubernetes/Rill)

With Kubernetes, you declare what you want, and the system will bring it to that state. This means the how-to is abstracted away, and you can focus on the what of your app.

On the other hand, using Airflow, you define every little step of the how inside your DAG; mixing business logic with technical boilerplate makes it inherently complicated and hard to decouple and get clarity in your runtime pipelines.

If you see Dagster, where you have “only” the business logic, what you want to do as part of your DAG, and everything else is outsourced into resources or Dagster Pipes, defining your compute with a simple annotation compute_kind="modal" or compute_kind="spark", not needing to care if it’s locally or on production, shows you the power of declarative with the side-effect of knowing your tools.

Comparing dbt with SQLMesh is another one, where SQLMesh understands plus parses the SQL with SQLGlot to get a semantic understanding of what the SQL does, giving it advances for powerful context-aware functions that dbt won’t be able to do.

Semantic Layers and BI tools define metrics within YAML (Cube, Rill), dashboards can be created entirely in Markdown (Evidence), and data pipelines are created on top of YAML (Kestra, Dagster).

Call it Infrastructure as Code, Visualization as Code, BI as Code, or anything as code. YAML engineering, or DSL (Domain Specific Language)—no matter what, it’s the way the data stack evolved to empower non-programmers to create data pipelines and infrastructure without losing the benefits of SW engineering best practices through declarative methods.

What Is a Declarative Data Stack?

A declarative data stack is a set of tools and, precisely, its configs can be thought of as a single function such as run_stack(serve(transform(ingest))) that can recreate the entire data stack.

Instead of having one framework for one piece, we want a combination of multiple tools combined into a single declarative data stack. Like the Modern Data Stack, but integrated the way Kubernetes integrates all infrastructure into a single deployment, like YAML.

We focus on the end-to-end Data Engineering Lifecycle, from ingestion to visualization. But what does the combination with declarative mean? Think of Functional Data Engineering, which leaves us in a place of confident reproducibility with little side effects (hopefully none) and uses idempotency to restart function to recover and reinstate a particular state with conviction or rollback to a specific version.

Why a Declarative Data Stack

The opposite would be an imperative data stack. The imperative way works well for simple homogenous systems. It is flexible and gives us a high level of control. However, it has the downside of a hard-to-manage state, so automatic integrity checks or failure management are needed.

But it’s not the right choice if you combine a heterogeneous data stack into a single data platform. The declarative approach manages complexity in a simple interface for non-technical users by abstracting away implementation details and focusing on the outcomes instead. It builds the foundation for a consistent, reproducible data platform across diverse components.

A declarative data stack cleanly separates semantics from implementation, decoupling the business code from technical implementation, the how is rendered by the data stack engine. This gives us a stateless stack that can be recreated from scratch with its configuration, setting aside intermediate data assets (stateful) produced along the way.

Think of Markdown. It is universally portable, and almost every application can render it with its various engines, such as HackMD, Obsidian, iA Writer, Neovim, and GitHub, to name a few. There is no need to fiddle with setting things up or configuring, as the Markup language defines everything—the engine decides/handles the rendering. The best part is that even images can be declaratively defined with Mermaid. Similarly, the declarative stack is programming language agnostic.

Let’s compare when to use which stack.

YAML: The Language of Declarative Configuration

YAML, Yet Another Markup Language, has become the configuration markup language for most modern tools. The reasons are simple: Compared to its predecessors, XML and JSON, which are still highly used, YAML is less verbose.

Image comparison between XML, JSON, and YAML

Besides their different reasons for use, XML is designed to support structured documents, while JSON should be simple and universal and can quickly be processed; JSON has been mainly used for small data sets and REST services. YAML has similar threats but tries to be a superset of JSON, although every JSON can effectively be a valid YAML file.

But why is YAML used for declarative configurations? It supports lists and dictionaries with almost no overhead. YAML is optimized for reading extended configurations. Its descriptive and portable structure across different programming languages and its clear interface make it easy to maintain, read, and modify, making it well-suited for this task.

YAML is also used when implementing a DSL (Domain-Specific Language) in your application. A DSL abstracts the complexity behind a system. It is a general-purpose language aimed at any software problem. Like Markdown or another example of HTML, it’s programming language agnostic; the engine, the browser, does not care how you generated the HTML; it knows what to do with it. That’s the goal of DSL, too.

Which Components Form an End-to-End Data Stack?

For it to be a complete data stack, we need to integrate data from its source systems, transform, aggregate, and clean data, and ultimately serve and visualize it, solving the core challenges in data engineering.

We cover all of these with the Data Engineering Lifecycle. Let’s go through them one by one.

Illustration of the data engineering lifecycle from the book Fundamentals of Data Engineering

Ingestion

Ingestion is the part that integrates the data sources into your data stack. Usually, your source types are OLTP databases, some files in a S3 bucket, or APIs.

Naturally, this is a very imperative process. You write a bunch of code, historically with procedural code or, in the modern world, with Python. But everything is writing the integration code, meaning the how. So, how do we get a more declarative approach? Wasn’t there a declarative language that is quite popular?

Yes, SQL. SQL is declarative. You say which columns (SELECT columns) from which table (FROM table) and with some criteria (WHERE) and some aggregations (GROUP BY). The rest is done by the SQL engine, typically the database, the Spark cluster, or anything else.

The most straightforward way is using the SQL approach and wrapping select * from s3://my-source-bucket/sales/*.parquet into an orchestrator or using something like dlt. dlt is, per se, not a declarative tool (except the REST API), but you can use it in a declarative fashion, e.g., configure a YAML to define all the columns and tables you want to ingest. Or you can use Airbyte and Dagster to create sources and destinations declaratively, essentially Data Integration as Code. Ibis is another lightweight, universal interface for data transformation; it uses Python and, therefore, is not declarative out of the box. However, you can interchange your SQL engine, too, while writing the code only once.

Transformation

The most significant layer of any data stack is the transformation or ETL part. As this is primarily done in SQL, we are fine here, right?

Not really. As SQL has limitations, no variables, lots of duplications, hard to define best practices, many have come to love dbt. dbt is a nice wrapper around SQL that gives you super powers through templating with Jinja and with goodies of documentation including lineage, versioning, and automating of your lost SQLs.

But with that, we are back to an imperative way. As dbt doesn’t know anything about its SQLs, it just runs them; it’s limited to doing anything data-aware.

Newer tools, especially SQLMesh and its open-source SQL parser SQLGlot, can analyze queries, traverse expression trees, and programmatically build SQL. We can also use them for declarative transformation. Especially with SQLMesh, we get a better understanding of semantics. This will give us the power to describe our transformations declaratively, even with live compiler checks, before running anything.

With that, it can automatically pre-detect breaking changes and run a missing backfill. The best thing is that SQLMesh is backward and compatible with dbt.

Serving

Visualizing and serving data in the correct format, whether as a data app, AI, or dashboard, is essential in your data stack. No matter how good your data quality and insights are, cleaning and transforming will be well-spent if you can present them in an easy-to-understand way.

Speaking of waste, the second wasteful time you can do is recreate the same dashboards repeatedly and change the measure on each so very slightly. This is where the power of declarative hits for visualization. Could you copy and paste the definition of a dashboard and replace the measure? That is precisely what we do with Rill or Evidence (interestingly with Markdown).

Here’s a declarative dashboard example in Rill:

This example, visualization as code, takes this declarative approach further, allowing us to define visualizations using structured data modeling syntax. The Grammar of Graphics is a related concept, a theoretical framework that breaks down statistical graphics into semantic components. Tools like ggplot2 in R and Vega-Light implement this grammar, enabling users to describe the relationships between data and visual elements rather than specifying how to draw the chart.

By embracing these declarative approaches to visualization, we can create more maintainable, flexible, and robust data presentations as part of our declarative data stack, focusing on what we want to communicate rather than the intricacies of how to draw it.

An excellent feature that can be integrated into a declarative approach is access permissions with access policies.

Undercurrents

In addition to these three main components of a data stack, there are some undercurrents across the data stack. We’ll focus only on three and keep it brief, as this could be another blog post.

Orchestration

Orchestration is the central piece that manages all moving parts. If you will, the orchestrator is the engine in which the how of your declarative data stack can be implemented. You can write the technical logic of fetching data from an API service, partition your data to optimize speed, and integrate different tools of your stack into a single data platform.

Orchestrators that already conform to a declarative way are Dagster, Kestra, KubeFlow, and many more. But as with browsers, you can have different engines, different orchestration tools, or even self-written code and small lambda functions. Even all this is up to you as the data stack engine architect.

Security

Security would greatly benefit from a declarative data stack, as you can configure your access rights at a central place and read that configuration for each of the components of your data stack instead of implementing it again at each layer.

DataOps

DataOps, as part of the data engineering lifecycle, could be seen as the place to manage your data stack engine. If we look at what DataOps is, a combination of efficiency, velocity, usability, and automation, this is precisely what the declarative data stack is about.

What About the Data Assets?

Before we wrap it up, let’s connect critical pieces at the interception where stateful and stateless meet: Data Assets, also called Data Products (e.g. in Data Mesh).

What are data assets? These are stateful assets that hold data. Assets can be a Parquet file on S3, a Delta/Postgres table, or a data mart on your OLAP cube.

This is the only part of the declarative data stack that cannot be configured or defined by a function, as upstream data is constantly changing and outside our control.

However, in an ideal data stack, we can always recreate data assets from source data if we perform a full/initial load every time. The moment we perform an incremental load, it gets more complex.

An interesting approach is the Software-Defined Asset, which adds declarative capabilities to your stateful assets—for example, defining to daily update the assets. That would check if downstream events have changed or automatically versioning data assets. These are all things you can implement and attach to your data assets, which makes even the data asset behave more declarative.

You could imagine writing a data stack engine for all data assets that automatically backfills required data assets if the declaration of your data stack has changed. All while keeping the idempotency and functional data engineering paradigm in mind.

This significantly simplifies data governance, as rules could be embedded in the assets and even automatically censor sensitive data. There is no need to add security rules, as data gets censored automatically. I could make countless more examples, but you get the point.

Durable State vs. Ephemeral State

We can further categorize the state into durable and ephemeral. The durable state, represented by our data assets, persists across system restarts and is crucial for maintaining long-term data integrity. On the other hand, the ephemeral state is temporary and often exists only during the execution of our declarative processes.

The challenge in a declarative system is managing the interaction between these state types. While our declarative configurations handle ephemeral states elegantly, the durable state (our data assets) requires special consideration. This is where concepts like Software-Defined Assets come into play, too, allowing us to apply declarative principles to durable state management.

Low Code vs. Code First

An interesting parallel is that this is also where low/no-code vs. code first comes from. Once you have a declarative data stack, you essentially have your low-code or no-code platform, as you can build an API/UI on top of your configs, and every user could change the data stack, assuming that the features are implemented and exposed as configuration.

Maslow’s Hierarchy of State in Declarative Data Stacks

Drawing inspiration from Maslow’s hierarchy of needs, we can conceptualize the hierarchies of state in declarative data systems in three levels, progressing from static to dynamic:

1. Basic Declarative State: The foundation consists of all declarations needed to recreate the data stack. This includes configurations, schemas, transformations, etc. It’s all static.
2. Asset-added State: Building on the basic state, we incorporate data assets. These can include source data, processed datasets, dashboards, reports, and other artifacts that represent the actual data and the state of the data stack.
3. Self-Actualizing State: The data stack becomes smarter and partially autonomous at the pinnacle. It can dynamically adapt to changes, such as automatically detecting new source data and generating appropriate declarative configs to integrate it into the entire stack. While the process remains deterministic based on the environment, it behaves dynamically.

As we move up this hierarchy, we increase the features and autonomy of our data stack, reducing manual intervention and increasing system “intelligence.”

Closed-source cloud data stacks usually achieve the self-actualizing state, as they have all the metadata and control the whole platform. But I believe this is also possible with an open declarative data stack. By centralizing the settings of the entire data stack in a single Git repository and ensuring end-to-end integration with comprehensive metadata, we can achieve similar levels of declarative deployment and setup. This approach allows for idempotent recreation of state across the entire data stack, mirroring the capabilities of closed-source platforms while maintaining openness and flexibility.

The Declarative Data Stack Engine

Let’s see what that could look like and what the data stack engine requires to implement. If we think in functional terms, we need to define the whole data stack with a single function:

1

run_stack(serve(transform(ingest)))

In an ideal scenario, we could run the whole stack declaratively with this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

run_stack(
    serve(
        template='github://covid/covid_dashboard.md',
        data=transform(
            {
                'groupby': 'country',
                'aggregate': {
                    'cases': 'sum',
                    'deaths': 'sum'
                }
            },
            ingest(
                source='duckdb',
                query='SELECT * FROM "s3://coviddata/covid_*.parquet"'
            )
        )
    )
)

This example:

1. Ingests data from a Parquet file in S3 using DuckDB
2. Transforms the data by grouping it by country and summing cases and deaths
3. Serves the transformed data using a dashboard template from GitHub

The whole definition and functions a declarative data stack would need to implement could look something like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

def run_stack(serve_function):
    return serve_function()

def serve(template, data):
    # Logic to render the template with the provided data
    pass

def transform(transform_function, data):
    return transform_function(data)

def ingest(source, query):
    # Logic to execute the query on the specified source
    pass

Note: I intentionally omitted detailed orchestration, as it could be the easiest way to implement such a data stack engine.

What’s Next?

In Summary, we need three things to recreate a declarative data stack from scratch:

1. Exogenous/external data sources
2. Declarative artifacts
3. Rendering engine

Integrating everything into one git repo for all common configs (and metadata) will allow us to achieve things similar to those of closed-source data platforms.

This is part one of a series. Next, in Part 2, we’ll explore the concept of “BI-as-code” and its effectiveness as an interface for GenBI (Generative Business Intelligence). We’ll begin with delving into why a declarative approach is a prerequisite for GenAI. We’ll also examine alternatives to the declarative data stack, providing a comprehensive view of the landscape. This part will highlight how the declarative paradigm is not just a trend but a fundamental shift in how we approach business intelligence in the age of AI.

Part 3 will take a practical turn as we attempt to build a declarative data stack example. Show how to implement this stack using AI, as declarative stacks allow a generative approach. The goal is to iterate and get one step closer to the open declarative data stack.

Further Reading

The Declarative Mindset in Data Stacks

1. Pedram Navid. “The Rise of the Data Platform Engineer.” Dagster Blog. An in-depth look at the evolving role of data platform engineers in modern data architectures. Read the article
2. Benoit Pimpaud. “ELT with Kestra, DuckDB, dbt, Neon and Resend.” Medium. A practical guide to implementing an ELT pipeline using various modern data tools. Read the article
3. Archie Wood. “An end-to-end data stack with just DuckDB: ETL is dead, long live ETV.” LinkedIn. Explores the concept of Extract-Transform-Visualize (ETV) using DuckDB as the primary tool. Read the post

Code Reusability in Data Transformation

4. Maxime Beauchemin. “Why Data Teams Keep Reinventing the Wheel: The Struggle for Code Reuse in the Data Transformation Layer.” Preset. Discusses the challenges and importance of code reusability in data transformation, introducing the concept of Parametric Data Pipelines. Read the article

Full article published at Rilldata.com - written as part of my services

In this third part, we’ll delve into data architecture patterns and their influence on data modeling. We’ll explore general and specialized patterns, debating the merits of various approaches like batch vs. streaming and lakehouse vs. warehouse, and the role of a semantic layer in complex data architecture.

We’ll also survey the landscape of data modeling tools, comparing commercial and open-source options, and ponder the potential of AI in data modeling. To wrap up, we’ll introduce data modeling frameworks like ADAPT™ and BEAM, designed to guide effective data model creation. Please join me as we take this exciting journey toward understanding data architecture better.

Data Architecture Pattern with Data Modeling

Bringing together the best of the individual approaches and techniques and knowing the common problems, we must always keep the bigger data architecture picture in mind. Sometimes you want a data vault modeling for your first layer when your source system is constantly changing, or you need a dimensional model in your last layer to build data apps on top of it or allow self-serve analytics. But how are you doing this?

For these, we need to look at data architecture per se. In this chapter, I will list some of the most common architectural patterns I’ve seen, but without going into all details of each.

General Purpose Data Architecture Pattern (Medallion, Core, etc.)

Let’s start with the one that suited me best, the one you have seen in one form or another. When I started at a consulting firm called Trivadis, and here it was called “Foundational Architecture of Data Warehouse”. We followed best practices such as staging>cleansing>core>mart>BI.

We would design these layers from the top-down approach discussed above and decide the data modeling technique for each layer depending on requirements.

Let’s have a detailed look at each layer, as these are fundamental for every data architecture project, and they will help us understand why we’d want to model different layers differently. The following layers or areas belong to a complete Data Warehouse (DWH) architecture but can be implemented into any data lake or analytics product you use.

Staging Area

Data from various source systems is first loaded into the Staging Area.

• In this first area, the data is stored as it is delivered; therefore, the stage tables’ structure corresponds to the interface to the source system.
• No relationships exist between the individual tables.
• Each table contains the data from the final delivery, which will be deleted before the next delivery.
• For example: In a grocery store, the Staging Area corresponds to the loading dock where suppliers (source systems) deliver their goods (data). Only the latest deliveries are stored there before being transferred to the next area.

Cleansing Area

It must be cleaned before the delivered data is loaded into the Core. Most of these cleaning steps are performed in the Cleansing Area.

• Faulty data must be filtered, corrected, or complemented with singleton (default) values.
• Data from different source systems must be transformed and integrated into a unified form.
• This layer also contains only the data from the final delivery.
• For example: In a grocery store, the Cleansing Area can be compared to the area where the goods are commissioned for sale. The goods are unpacked, vegetables and salad are washed, the meat is portioned, possibly combined with multiple products, and everything is labeled with price tags. The quality control of the delivered goods also belongs in this area.

Core

The data from the different source systems are brought together in a central area, the Core, through the Staging and Cleansing Area and stored there for extended periods, often several years.

• A primary task of the Core is to integrate the data from different sources and store it in a thematically structured way rather than separated by origin.
• Often, thematic sub-areas in the Core are called “Subject Areas.”
• The data is stored in the Core so that historical data can be determined at any later point in time.
• The Core should be the only data source for the Data Marts.
• Direct access to the Core by users should be avoided as much as possible.

Data Marts

In the Data Marts, subsets of the data from the Core are stored in a form suitable for user queries.

• Each Data Mart should only contain the data relevant to each application or a unique view of the data. This means several Data Marts are typically defined for different user groups and BI applications.
• This reduces the complexity of the queries, increasing the acceptance of the DWH system among users.
• For example, The Data Marts are the grocery store’s market stalls or sales points. Each market stand offers a specific selection of goods, such as vegetables, meat, or cheese. The goods are presented so that they are accepted, i.e., purchased, by the respective customer group.

Not every architecture is the same

Only some data warehouses or data engineering projects have precisely this structure. Some areas are combined, such as the Staging and Cleansing areas, or differently named. The Core is sometimes referred to as the “Integration Layer” or “(Core) Data Warehouse.”

However, the overall system must be divided into different areas to decouple the other tasks, such as data cleaning, integration, historization, and user queries. In this way, the complexity of the transformation steps between the individual layers can be reduced.

Why use it today?

Isn’t it amazing that something from 2016 is still so current? That’s why data modeling is getting into vogue again, as it has never been entirely outdated.

Databricks renamed these layers with bronze, silver, and gold to understand it may be a little better and called it Medallion Architecture, but it’s something every BI engineer works with every day. In essence, it’s the same concept.

Or if we look at the Data Engineering Lifecycle introduced by the Fundamentals of Data Engineering, we see a similar picture but on an even higher level. You could apply the different layers to the Storage layer in the image below.

Specialized Data Architecture Patterns

In this chapter, we look at patterns that might not be considered in formal data modeling, but each of its decisions will highly influence the data modeling part. Besides the general data architecture, I call these specialized data architecture patterns as these are higher-level data architecture decisions.

Batch vs. Streaming

An obvious decision you need to take early on is if you need real-time data for critical data application or batch with near real-time micro batching every minute, the hour is enough.

Still, to this day, steaming is primarily optional. Suppose you tell the business team you can achieve near real-time with hourly batching; they will be happy. The latest up-to-date data will little influence your data analytics. Most cases are looking at historical data anyway.

Nevertheless, some business-critical data solutions need real-time. However, be aware that the effort and challenges will be much more significant as you can only partially recover if a stream fails. But the good idea is to set up your data pipeline as even based on the streaming approach, and therefore can go lower and lower with the latency of your batch to achieve near real-time.

Data Lake/Lakehouse vs. Data Warehouse Pattern

As Srinivasan Venkatadri says correctly: “Data modeling, in general, should also talk about files (open formats), Lakehouses, and techniques to convert or extract structural data from semi or unstructured would help.” This is precisely where data lake/lakehouse or data warehouse patterns apply. With these, you need to think about these questions.

With the long fight between ELT with Data Lakes and traditional ETL with Data Warehouses, the different architectures greatly influence the data modeling architecture.

I will only go into some detail here, as I have written extensively about these topics. But the decision to go with a data lake or lakehouse will likely influence the data modeling to a more open source strategy and tools. Also, as you are dumping files into your lake, you need to work with large distributed files, where Table Format makes a lot of sense, to get database-like features on top of files.

Semantic Layer (In-memory vs. Persistence or Semantic vs. Transformation Layer)

The data lake decision is also highly influential with the new Rise of the Semantic Layer, where you run a logical layer on top of your files in the case of a data lake and the possibility of a data warehouse on top of your data marts.

This is an interesting one, as we talked in part 1 and part 2 about the Conceptual, Logical, and Physical Data Models where a Semantic Layer could replace the logical model. The semantic layer stores the queries in declarative YAMLs, and data not persisted in physical storage is executed at run time or when the data is fetched. Integrating the semantic layer into complex data architecture and overall data modeling makes sense. It can serve a wide range of data consumers with different requirements, such as direct SQL, API (REST), or even GraphQL, with the great benefit of writing business logic only once. Here again, as business logic is the biggest treasure of data modeling, we have many new options to model our data with semantic layers.

The counter architecture pattern is to go with a transformation layer that typically can be achieved with dbt or a Data Orchestrator, where you persist each step into physical storage. Mainly to gain faster speed querying these data sets and reusing the data sets with other marts or data apps. The transformation layer and its transformational modeling make a decision between table formats vs. in-memory formats, query push-downs, and data virtualization.

Modern/Open Data Stack Pattern

Next up is Modern Data Stack architecture, or Open Data Stack. This pattern is basically for sheer choice we have nowadays to choose from the open data stack. It is its architecture to choose the right tool best suited for the requirements at hand in the company.

It matters if you chose an open source tool, v0.5, that might get out of order in a couple of years or if you chose the v0.1 that made the crackdown the line. But these are complicated bets, and only experienced data architects and people working in the field can make decisions based on intuition for the right way a project is progressing.

It is also to say no to yesterday’s new shiny tool, resist the urge, and wait until the product is more mature. But at the same time to take risks and bet on a tool that is open to success and embraces open-source philosophy instead of building a worse copy in-house. Each enterprise company either uses a closed-source solution to handle the data engineering lifecycle or makes its stack or newer option, betting on a developed framework in the open.

Imperative vs. Declarative Pattern

The declarative approach is manifesting itself more and more. Started within the fronted revolution where react was declaring components, Kubernetes did it for DevOps, Dagster revolutionized orchestration with Software-Defined Assets or us at Airbyte, where we created a Low-Code Connector developer kit to create data integrations in minutes by filling out a YAML file.

It is only possible because of drastically reduced complexity and the need to write many boilerplates. The declarative way describes what and the how is taken care of in the framework.

I have much more to say, but you can read more on our data glossary on declarative and how it relates to functional data engineering and what the opposite, imperative pattern is.

Notebooks vs. Data Pipelines Pattern

A pattern directly related to orchestration is the notebooks versus data pipeline pattern, where you can write and run a pipeline solely a notebook or a mix of mature, unit-tested data pipelines with a stable orchestrator with all bells and whistles included.

Centralized and Decentralized Pattern

Suppose you are in the field for a while. How many cycles have you been through from doing everything server side, switching everything to server rendering, and forth and back with the battle of client vs. server, microservices vs. monoliths, and lately, central cloud data warehouses vs. a yet-to-show decentralized Data Mesh. And many more forth and back will follow.

Whatever you choose, start with simplicity. You can always add complexity later when the product and solution mature.

Everything else

There are so many more patterns I will write about but at some time later. For now, I leave you with pointers above, and I’ll come back to it sometime later.

Data Modeling Tools

Popular data modeling tools include Sqldbm, DBDiagrams, Enterprise Architect, and SAP PowerDesigner. These tools are widely used in the industry and offer powerful features such as data modeling, profiling, and visualization.

Open-source data modeling tools such as MySQL Workbench and OpenModelSphere are free and offer essential features for creating data models. They are helpful for small projects and provide an opportunity for data engineers to learn data modeling skills.

Choosing the right data modeling tool depends on the organization’s needs, budget, and project size. Large organizations may require expensive enterprise-level tools, while small businesses may opt for open-source tools. Selecting a tool that is easy to use, has the needed features, and is compatible with the organization’s database management system is essential.

Other tools are Ellie.ai, whose key features are Data Product Design, Data Modeling, Business Glossary, Collaboration, Reusability, and Open API.

dbt can be seen as a transformation modeling tool. Dagster can be used as a DAG modeling tool. And so forth. But you can also use ExaliDraw for Markdown-based drawing or draw.io (lots of templates for AWS, Azure, etc.) to draw architectures.

If you struggle to think in dbt tables and SQL is not the SQL is not the right language. One problem, SQL is a declarative language, which is a blessing and a curse. Especially if you do recurring queries, SQL gets nasty spaghetti coded, which again dbt helps with Jinja Templates, but as it’s not a language, without much in-built support. Reconfigured (not free) was built for people without years of experience, focusing heavily on business logic.

What about ChatGPT?

With all the hype of generative AI, specifically ChatGPT, it asks if AI can model our data.

If we recap the information from this series, most of it condenses into translating business requirements into a data model or semantics, also called business logic. As Chad Sanderson mentioned in his post:

The hard part of data development is understanding how code translates to the real world. Every business has a unique way of storing data. One customer ID could be stored in a MySQL DB. Another could be imported from Mixpanel as nested JSON, and a third might be collected from a CDP. All three IDs and their properties are slightly (or significantly) different and must be integrated.

As Chad continues, and I strongly agree, he says, “As smart as ChatGPT might be, it would need to understand the semantics of how these IDs coalesce into something meaningful to automate any step of modeling or ETL. The algorithm must understand the real world, grok how the business works, and dynamically tie the data model to that understanding.”

This is not AI or ML anymore; that is more intuition, long-term experience in working in the field, and experiencing some of the challenges to get a feeling for it.

If there is a place where you do not want to use automation, then it would be in data modeling and the overall data architecture. Here we need discussions and a deep understanding (domain knowledge) of the field and data modeling.

On the other hand, based on your data model, you can use AI to generate schemas or the physical model of a database. Brainstorm with ChatGPT based on your created data model if something is missing or if it sees something that can be changed. Sometimes you get great insights from it by providing the solution you came up with.

Maybe most importantly, a data engineer will always be needed to assess the outcome, understand and make sense of the data in front of him and understand the overall data flow of the organization. That can only be delegated for a while.

Data Modeling Frameworks

Besides tools, there are also helpful frameworks that help you model your data, asking the right questions.

ADAPT™

ADAPT says that more than existing data modeling techniques like ER and dimensional modeling is required for OLAP database design. That’s why ADAPT is a modeling technique designed specifically for OLAP databases. It addresses the unique needs of OLAP data modeling. The basic building blocks of ADAPT are cubes and dimensions, which are the core objects of the OLAP multidimensional data model.

Although ADAPT was created for OLAP cubes in the old days, most techniques and frameworks also apply to regular data modeling nowadays. There are nine ADAPT database objects, and their symbols illustrate how to use logos with simple examples.

Why ADAPT over ER and dimensional modeling

ADAPT is considered superior to ER and dimensional modeling for several reasons:

1. Incorporation of Both Data and Process: ADAPT incorporates both data and process in its approach, which is particularly useful for designing (OLAP) data marts.
2. Logical Modeling: ADAPT emphasizes logical modeling. This prevents the designer from jumping to solutions before fully understanding the problem.
3. Enhanced Communication: ADAPT enhances communication among project team members, providing an everyday basis for discussion. This improved communication leads to higher-quality software applications and data models.
4. Comprehensive Representation: ADAPT allows for the representation of an (OLAP) application in its entirety without compromising the design due to the limitations of a modeling technique designed for another purpose.

In summary, ADAPT is a says to be a more flexible, comprehensive, and communication-enhancing modeling technique for OLAP databases compared to ER and dimensional modeling.

BEAM for Agile Data Warehousing

BEAM, or Business Event Analysis & Modeling, is a method for agile requirement gathering designed explicitly for Data Warehouses, created by Lawrence Corr and Jim Stagnitto in the Agile Data Warehouse Design book. BEAM centers requirement analysis around business processes instead of solely focusing on reports.

It uses an inclusive, collaborative modeling notation to document business events and dimensions in a tabular format. This format is easily understood by business stakeholders and easily implemented by developers. The idea is to facilitate interaction among team members, enabling them to think dimensionally from the get-go and foster a sense of ownership among business stakeholders.

The principles of BEAM include:

• Modelstorming: Business intelligence is driven by what users ask about their business. The technical setting is secondary. Storyboarding the data warehouse to discover and plan iterative development
• Asking stories with 7W: Telling dimensional data stories using the 7Ws (who, what, when, where, how many, why, and how—model by example, not abstraction; using data story themes.
• Visual modeling: Sketching timelines, charts, and grids to model complex process measurement – simply
• Business Driven: Well-documented data warehouses that take years to deploy will always need to be updated. Business users will look elsewhere. Usually with business units: “I need it now, or I’d rather stick with Excel solution ….”
• Customer Collaboration: End users’ business knowledge is your greatest resource.
• Responding to Change: Promoting change through the above actions, leading to weekly delivery cycles.
• Agile design documentation: Enhancing star schemas with BEAM dimensional shorthand notation

Lawrence Corr emphasizes the importance of asking the right questions or “data stories.” For instance, a customer’s product purchase could trigger questions about the order date, purchase and delivery locations, the quantity bought, purchase reason, and buying channel. A comprehensive picture of the business process is formed by carefully addressing these questions and providing the basis for technical specifications.

Common Data Model

There are examples of standard data models, so you do not need to start from scratch. The concept behind these approaches is to transform data contained within those databases into a standard format (data model) and a common representation (terminologies, vocabularies, coding schemes), then perform systematic analyses using a library.

For example, every model needs dimensions such as customer, region, etc. Some references I found you see below:

• Microsoft started The Common Data Model (CDM).
• OMOP Common Data Model for health care. It allows for systematically analyzing disparate observational databases.
• Elastic has its Common Schema (ECS) Reference.
• A bit more advanced, but a DBML (Database Markup Language), an open-source DSL language designed to define and document database schemas and structures, tries to create a standard for determining these models. DBML is intended to be simple, consistent, and highly readable. Works well with dbdiagram.io.

Applying Data Modeling / Best Practices?

At the end of all we learned, how do you apply data modeling in practice? This series gave you a good introduction and some pointers to look for when you start with data modeling.

Below I found the Best Practices for Data Modeling, which guides you through some of the critical steps from one, designing the data model to eleven, verifying and testing the application of your data analytics:

1. Design the data model for visualization
2. Recognize the demands of the business and aim for relevant results
3. Establish a single source of truth
4. Start with simple data modeling and expand later
5. Double-check each step of your data modeling
6. Organize business queries according to dimensions, data, filters, and order
7. Perform computations beforehand to prevent disputes with end customers
8. Search for a relationship rather than just a correlation
9. Using contemporary tools and methods
10. Enhanced data modeling for improved business results
11. Verify and test the application of your data analytics

Another excellent best practice for dbt can be found in How to Write a High-Quality Data Model From Start to Finish Using dbt by Madison or applying dimension model with Kimball and dbt in Building a Kimball dimensional model with dbt (GitHub) by Jonathan.

The Future of Data Modeling

As I’ve delved into the intricacies of data modeling in the previous parts of this series, it’s clear that we’re witnessing a revolution in the way we perceive, manage, and interact with data. The digital age is characterized by information overload, and data modeling provides the framework to harness this data and transform it into valuable insights.

Reflecting on the future of data modeling, I can’t help but feel a sense of optimism mixed with anticipation. It’s thrilling to envision a world where data-driven decisions are the norm rather than the exception, and I genuinely believe we’re on the right track.

Emerging technologies like AI and machine learning promise to streamline further and automate the process of data modeling. There’s potential for AI to take on a more active role in data modeling, translating complex business logic into coherent data structures.

This vision, however, doesn’t mean we can become complacent. It’s more crucial than ever for data professionals to stay on top of evolving industry trends and techniques. And then, there’s the matter of the various data modeling tools available. The future will likely expand the number of open-source and proprietary options. But at the end of the day, selecting the right tool will always come down to your specific requirements, constraints, and the nature of your data.

We also must remember the importance of data architecture patterns. As data grows in volume and complexity, finding the most suitable architecture becomes increasingly critical. The choice between batch vs. streaming or data lake vs. data warehouse could significantly impact your data modeling efforts. So decisions around implementing a semantic layer, opting for a modern/open data stack, or navigating between centralized and decentralized patterns.

As I wrap up this series of data modeling, I encourage you all to keep learning, experimenting, and pushing the boundaries of what’s possible with your data. Remember, the essence of data modeling is simplicity, no matter how complex the underlying data might be. The future of data modeling is constant change. But one is inevitable; it will be critical for every company. I’m excited to see how the field evolves and how we, as data practitioners, continue to drive this evolution :).

Learning more about Data Modeling

Below are some resources and helpful comments I gathered from you all; thank you for the valuable feedback throughout writing this article.

Resources

• MongoDB Courses and Trainings, offering comprehensive guides to understanding and mastering MongoDB. Link
• Book: “Agile Data Warehouse Design: Collaborative Dimensional Modeling, from Whiteboard to Star Schema” by Lawrence Corr. This is a foundational text for understanding dimensional modeling in an agile context. Link
• Book: “Data Warehouse Blueprints: Business Intelligence in der Praxis” by Dani Schnider, Claus Jordan, Peter Welker, and Joachim Wehner. This is an excellent resource for German speakers. Link
• Book: “Data and Reality” - a timeless guide to data modeling, recommended by Jenna Jordan.
• Video: “Data Modeling in the Modern Data Stack” - a valuable resource for understanding the current state of data modeling. Link
• Article: “Introducing Entity-Centric Data Modeling for Analytics” on Preset - a good read for understanding an entity-centric approach to data modeling. Link
• Website: AgileData.io by Shane Gibson - a resource for reducing the complexity of managing data for Leaders, Analysts, and Consultants. Link
• Podcasts: “Shane Gibson - Making Data Modeling Accessible - The Joe Reis Show” on Spotify. Link

Helpful comments

• Use well-defined ontologies that describe your business and relationships between components using common industry concepts, as suggested by Rayner Däppen. Link to comment
• Keep your data models updated, aligned, documented, validated, and verified with the business. This will ensure the models accurately reflect the current state of the company.
• Consider where to build the semantic/metrics layer to allow for fast, interactive analytics/dashboards and how to make it available to multiple tools to avoid various definitions. Link to comment
• From “Data Modeling in the Modern Data Stack” - computation is now the expensive part of data modeling, not storage. A hybrid approach is often used in modern data stacks to balance complexity, computational cost, data redundancy, and adaptability.

Originally published at Airbyte.com

As we delve deeper into this topic, Part 2 will focus on data modeling approaches and techniques. These methods play a vital role in effectively designing and structuring data models, allowing organizations to gain valuable insights from their data.

We will discuss various data modeling approaches, such as top-down and bottom-up, and specific data modeling techniques, like dimensional modeling, data vault modeling, and more. We will address the common challenges faced in data modeling and how to mitigate them. By understanding these approaches and techniques, data engineers can better navigate the complexities of data modeling and design systems that cater to their organization’s specific needs and goals.

Data Modeling Approaches

When designing your data model, you typically begin with a top-down approach. You sit with your business owners and domain experts and ask them questions to understand what entities your organization will implement, e.g., customer, product, and sales. Bottom-up is the alternative, where you go from a physical data model. We’ll discuss it in a minute.

Agreeing on how you visualized your data flow, and your data is essential to brainstorm up front between domain experts and the data engineering, business intelligence, or analytics engineer involved. It will help you design it in steps, avoid siloed modeling by engineers only, and define familiar entities used company-wide.

It also helps and should be done to define KPIs and Metrics upfront. These are agreed-upon goals you want to achieve together. As a data modeler, here is where you get the dimensions you need and the fact granularity. E.g., are we talking monthly, weekly, or daily revenue numbers? Are these plotted on a map per city or country?

The challenge of data modeling is Data Literacy. Data literacy is the ability to derive meaningful information from data, just as literacy, in general, is the ability to derive information from the written word—or, said differently, extrapolating the business value from the data given. Let’s figure out how to mitigate those problems with processes and techniques.

Conceptual, Logical, and Physical Data Models

Let’s start with the Conceptual Data Model represents a high-level view (top-down), the logical data model provides a more detailed representation of data relationships, and the physical data model defines the actual implementation in the database or data storage system (bottom-up)—more on the top and bottom-up approaches in the following chapters.

A common approach is to start with the conceptual model, where you define the entities in your organization from a top-down and high-level perspective and model them together. Usually, the Entity Relationship Diagram (ERD) is used for this.

Later, you move to a logical model where you add more details, such as the ID, whether you create a PK or use the business key from the source system. Are you modeling the customer as normalized tables (customer, address, geographic), or do you keep it simple with duplications? How do you implement change history of data; are you using Slowly Changing Dimension (Type 2) or snapshotting all dimensions?

The physical model is where you implement or generate it to the destination database system respecting each database’s slightly different syntax.

The Benefit of a conceptual model in a larger enterprise organization of sufficient complexity has a conceptual data model gives you a great framework to order the logical and physical models around, decoupled from the source systems.

Conceptual Data Model: Top-Down Approach

In a top-down approach, you start with a high-level view of the organization’s data requirements. This involves working with business owners, domain experts, and other stakeholders to understand the business needs and create a conceptual data model.

Then, you iteratively refine the model, moving from conceptual to logical and finally to the physical data model. This approach is particularly suitable when there is a clear understanding of the business requirements and goals.

Physical Data Model: Bottom-Up Approach

On the other hand, the bottom-up approach begins with analyzing the existing data sources, such as databases, spreadsheets, or different structured and unstructured data. Based on this analysis, you create a physical data model that reflects the current data storage and relationships.

Next, you work your way up, creating a logical data model to represent the business requirements and a conceptual model to provide a high-level view of the data. The bottom-up approach is beneficial when dealing with legacy systems or when there needs to be more knowledge of the organization’s data requirements.

Combining top-down and bottom-up approaches in data modeling

Combining top-down and bottom-up approaches may be the best solution in many cases. By blending these methods, you can capitalize on the strengths of each approach and create a comprehensive data model that meets your organization’s needs.

Regardless of your chosen approach, it’s essential to maintain clear communication among all stakeholders and ensure that the data model aligns with the organization’s objectives and supports effective decision-making.

Hierarchical Data Modeling, Network Data Modeling and Object-Role Modeling?

While searching for other approaches, I came across hierarchical, network, and object-oriented data modeling.

The hierarchical data modeling organizes data in a tree-like structure, with parent-child relationships between entities. It is suitable for representing hierarchical data or nested relationships, such as organizational structures or file systems.

The network data modeling approach models data as interconnected nodes in a graph, allowing for complex relationships between entities. It helps represent many-to-many relationships and networks, such as social networks, transportation networks, or recommendation systems.

The object-role modeling is an attribute-free, fact-based data modeling method that ensures a correct system and enables the derivation of ERD, UML, and semantic models while inherently achieving database normalization.

Some more, such as the flat model, object–relational model, are listed on Wikipedia for data models.

Data Modeling Techniques

In Part 1, we introduced the various levels of data modeling, including generation or source database design, data integration, ETL processes, data warehouse schema creation, data lake structuring, BI tool presentation layer design, and machine learning or AI feature engineering. We also discussed different approaches to data modeling in the previous chapter. This chapter will delve deeper into the practical techniques used in the data modeling process.

These techniques are primarily employed in batch-related processes and cater to the design and modeling of Data Warehouses, Lakes, or Lakehouses. We will explore each technique’s unique benefits and applications in modern data engineering.

Dimensional Modeling

There are many different techniques, but Dimensional Modeling is probably the most famous and the one that has stood out the longest. Its birth was with the inception of the data warehouse and the release of the iconic The Datawarehouse Toolkit book in a 1996 book.

The data space has changed a lot since then, so the question arises, “Is dimensional modeling still needed within data engineering compared to its popularity way back?” Let’s find out in the following chapters.

A quick reminder of how data modeling looked for a long time:

Data Modeling vs. Dimensional Modeling

Let’s start with the difference between dimensional and data modeling to understand why we even discuss it.

• Data modeling is the broad term that encompasses various techniques and methodologies for representing and modeling data across a company.
• Dimensional modeling is a specific approach to data modeling that is particularly suited for data warehousing, business intelligence (BI) applications, and newer data engineering data models.

What is Dimensional Modeling

So what, then, is dimensional modeling? Dimensional modeling focuses on creating a simplified, intuitive structure for data by organizing data into facts and dimensions, making it easier for end-users to query and analyze the data.

In dimensional modeling, data is typically stored in a star schema or snowflake schema (more later), where a central fact table contains the quantitative data, and it is connected to multiple dimension tables, each representing a specific aspect of the data’s context. This structure enables efficient querying and aggregation of data for analytical purposes.

The dimensional modeling approach focuses on identifying the key business entities and modeling these in an easy-to-understand way for consumers.

With the DW/BI Lifecycle Methodology that was created later in the 90s, Kimball’s core ideas applying still to this very day which is:

• Focus on adding business value across the enterprise.
• Dimensionally structure the data that are delivered to the business.
• Iteratively develop in manageable lifecycle increments rather than attempting a Big Bang approach.

Key concepts of dimensional modeling could be an article, and so much content exists. I leave you here with some links to learn more. For example, around dimensions with Conformed Dimensions, Junk Dimension or Slowly Changing Dimension, Additive, semi-additive, and non-additive facts, and many more; check out Dimensional Modeling Techniques for more.

Why is Dimensional Modeling Still Relevant Today?

But is dimensional modeling and all its associated concepts still relevant today? The answer is a resounding yes—perhaps even more so than before. As you’ve read in the preceding chapters, dimensional modeling aims to achieve a focus on business value. In today’s rapidly evolving world, this crucial aspect is sometimes overlooked.

By incorporating a robust dimensional model at the core of every data project, data engineers are compelled to consider critical questions related to granularity, entities, metrics, and more. Addressing these essential aspects upfront and working towards them is invaluable for achieving business goals and driving project success.

Star Schema vs. Snowflake Schema: Kimball and Inmon

In data warehousing, the star and snowflake schemas are standard data modeling techniques that are highly related to dimensional modeling.

The star schema, typically associated with Kimball’s approach, has a central fact table connected to dimension tables, emphasizing staging and denormalized core data. This bottom-up approach allows for quicker access to denormalized data for analysis. Conversely, the snowflake schema further normalizes dimension tables, creating a more complex structure. This schema aligns with Inmon’s approach, emphasizing a highly normalized core close to the source system, suitable for large-scale data warehousing projects.

The nuances between the star and snowflake schemas and their corresponding approaches can guide data professionals in designing and implementing data warehouses, but won’t dictate success or failure. Both schemas have their merits, and their choice depends on personal preference or specific project requirements. Understanding these differences allows data professionals to make informed decisions tailored to their unique projects.

Data Vault Modeling: A Flexible and Dynamic Approach

Compared to dimensional modeling, Data Vault modeling is a method that addresses the challenges of modern data warehousing, mainly when dealing with big data and fast-changing data connection points. This hybrid data modeling approach combines the best aspects of 3NF (Third Normal Form) and Star Schema methodologies, resulting in a scalable, flexible, and agile solution for building data warehouses and data marts.

The primary components of a Data Vault model are Hubs, Satellites, and Links. Hubs represent business keys, Satellites store descriptive attributes, and Links define relationships between Hubs. This unique structure allows for rapid ingestion of new data sources, supports historical data tracking, and is well-suited for large-scale data integration and warehousing projects.

Moreover, Data Vault modeling has been increasingly utilized as a governed Data Lake due to its ability to adapt to changing business environments, support massive data sets, simplify data warehouse design complexities, and increase usability for business users. By modeling after the business domain, Data Vault ensures that new data sources can be added without impacting the existing design. As a result, Data Vault modeling, in conjunction with Data Warehouse Automation, is proving to be a highly effective and efficient approach to data management in contemporary data-driven business landscapes.

Anchor Modeling

Anchor Modeling is an agile data modeling technique designed to handle evolving data structures like data vault modeling. It is built around storing each attribute as a separate table, allowing for more flexibility when dealing with schema changes. This approach is beneficial when the data model must frequently evolve and adapt to new requirements. Anchor Modeling is known for efficiently handling schema changes and reducing data redundancy.

Compared to data vault modeling, anchor modeling focuses on the change in information both in structure and content. It separates identities (anchors), context (attributes), relationships (ties), and finite value domains (knots). The focus is on the flexibility and temporal capabilities of the data model, capturing changes in information over time.

Bitemporal Modeling: A Comprehensive Approach to Handling Historical Data

A more niche but still valid modeling technique is Bitemporal Modeling.

Bitemporal modeling is a specialized technique that handles historical data along two distinct timelines. This approach enables organizations to access data from different vantage points in time. It allows for the recreation of past reports as they appeared and how they should have appeared, given any corrections made to the data after its creation. Bitemporal modeling is beneficial in sectors like financial reporting, where maintaining accurate historical records is critical.

Focusing on the completeness and accuracy of data, bitemporal modeling allows for creating comprehensive audit trails. Using bitemporal structures as the fundamental components, this modeling technique results in databases with consistent temporality for all data. All data becomes immutable, enabling queries to provide the most accurate data possible, data as it was known at any time and information about when and why the most accurate data changed.

Bitemporal modeling can be implemented using relational and graph databases, and while it is different from dimensional modeling, it complements database normalization. The SQL:2011 standard includes language constructs for working with bitemporal data. Read more on a gentle introduction to bitemporal data challenges.

Entity-Centric Data Modeling (ECM)

A relatively new modeling technique is Entity-Centric Data Modeling (ECM) introduced by Maxime Beauchemin. Entity-centric data modeling (ECM) elevates the core idea of an “entity” (i.e., user, customer, product, business unit, ad campaign, etc.) at the very top for analytics data modeling.

It’s interesting as its core focuses on the strength of precisely the points discussed above that dimensional provides. As it’s old, it also has some missing features that today’s world is needed. That’s why Max updated it and merged it with Feature Engineering, used in the ML project. You can find more in his latest article or my comments in the DataNews.filter() newsletter 📰.

Common Problems with Data Modeling

Data modeling is easy to neglect; assessing the consequences can take time and effort. The image below illustrates that if you initially ignore poor data modeling and architecture decisions, you’ll likely notice problems in the last mile, thinking they might be due to the tools or insights. However, the fundamental issues primarily originate in the first part of the data analytics cycle.

Here are some critical data modeling problems.

Business rules, which are specific to how things are done in a particular place, are often embedded in the structure of a data model. This leads to a problem where small changes in business processes result in significant changes in computer systems and interfaces. To address this issue, business rules should be implemented flexibly, avoiding complex dependencies and allowing the data model to adapt efficiently to changes in business processes.

Another common issue is that entity types are often not identified or incorrectly identified. This can cause data replication, duplicated data structures, and functionality. These duplications increase the costs of development and maintenance. Data definitions should be explicit and easy to understand to prevent this problem, minimizing misinterpretation and duplication.

Data models for different systems can vary significantly, creating a need for complex interfaces between systems that share data. These interfaces can account for between 25 and 70% of the cost of current systems. To address this, required interfaces should be considered inherently during data model design, as data models would only be used independently with interfaces within different systems.

Lastly, data cannot be shared electronically with customers and suppliers because the structure and meaning of data still need to be standardized. To maximize the value of an implemented data model, it is crucial to define standards that ensure data models meet business needs and maintain consistency. This standardization will enable efficient data sharing between various stakeholders.

To mitigate these issues, tight integration into the overall data architecture and patterns can reduce friction. We’ll explore these in the next part, 3.

What’s Next in the Last Part?

Throughout Part 2, we have explored the various data modeling approaches and techniques that serve as the backbone of data engineering. From top-down and bottom-up approaches to conceptual, logical, and physical data models, understanding these methods is crucial for effective data modeling. Techniques like dimensional, data vault, and bitemporal modeling offer unique benefits and cater to a wide range of use cases in modern data engineering. As we have seen, addressing common problems in data modeling and ensuring tight integration into the overall data architecture is essential for success.

In the next Part of this series, “Data Architecture Patterns, Tools, and The Future—Part 3”, we will delve into data architecture patterns, tools, and the future of data modeling. Stay tuned as we explore the fascinating world of data modeling within data engineering and its impact on the future of data-driven decision-making.

Originally published at Airbyte.com