Compute and Storage Separation

The compute-storage separation model has become increasingly popular in cloud OLAP systems because it allows for independent scaling of resources and often supports features like storage-based billing and compute-based billing separately.

The opposite of shared-nothing architecture, Compute and Storage Separation:

• Compute and storage layers are decoupled
• Storage is centralized and accessible by all compute nodes
• Compute resources can scale independently from storage
• Enables more flexible resource allocation and cost optimization
• Examples: Snowflake, Google BigQuery, Amazon Redshift Spectrum, ClickHouse Cloud

Architectures such as Druid or ClickHouse using co-location feature:

Even though Druid has different node types handling different aspects of data processing, it still follows the co-located principle where the nodes processing queries have direct, high-speed access to the data they need. The memory-first approach with disk as a fallback is fundamental to their high-performance architecture.

So yes, despite having different components or machines handling different aspects of processing, both ClickHouse and Druid are correctly categorized as using co-located compute and storage architecture, particularly from the perspective of query performance and how data is accessed during processing.

The opposite is decoupled cloud storage + compute architectures like Cloud Data Warehouses.

The main categories, decoupled cloud storage + compute architectures for Cloud Data Warehouses such as Snowflake etc, or Compute and Storage Separation like Modern OLAP Cubes.

I write more on on Scaling Beyond Postgres: How to Choose a Real-Time Analytical Database (internal).

Yury Izrailevsky and Jordan Tigani were saying that the speed will never be as fast as OLAP or DWH when compute and storage is decoupled ^812871

It’s interesting what StarRocks is doing with reading Iceberg and converting to its native format—same approach as StarTree with Pinot and ClickHouse. But again, it will never be as fast, and there will always be trade-offs.

Regarding Druid and ClickHouse, you’ve touched on an important aspect:

1. ClickHouse: ClickHouse does keep data and compute together on the same nodes. Data is stored in a column-oriented format directly on the nodes that process queries, allowing for very fast access. Its vectorized query execution further optimizes performance by processing blocks of data at once.
2. Druid: Druid has a more nuanced architecture:
   • It separates real-time and historical data processing
   • Real-time nodes ingest and process recent data, keeping it in memory for fast access
   • Historical nodes serve older data that has been moved to deep storage

• Compute Engine
• Open Data Platform
• Data Engineering Architectures Overview

Origin: OLAP Comparison - Rill, and later 2025-05-20: shared-nothing architecture
References: Co-located storage principle
Created 2025-03-03