Metrics Layer

Last updated Feb 9, 2024

Similarities to a Semantic Layer

The metrics layer is a key component of a Semantic Layer. Often, a basic metrics layer is integrated within a BI tool, translating its metrics specifically for that BI tool.

Also known as a metrics store or headless BI, the metrics layer encompasses a detailed specification of metrics like Measures and Dimensions. It typically involves model parsing from files, predominantly YAML, and APIs to develop and implement metric logic, often including a cache layer. This approach advocates for the DRY (Don’t Repeat Yourself) principle, allowing for single-time definition and subsequent deployment across various BI tools or internal applications and processes.

Discover more on Semantic Layer or The Rise of the Semantic Layer.

An excellent depiction illustrating the placement and role of your Metrics Layer | Emerging Architectures for Modern Data Infrastructure - a16z

Another insightful perspective RW Down the Semantic Rabbit Hole - By JP Monteiro:

The essence of a semantic layer is to bridge the gap between “business language” and “data language”, offering a unified, consistent business domain representation in the data.

“Metrics Stores” – emerging as a pivotal element in the data stack, centralizing key business metrics Data & AI Landscape 2021

In essence, the Headless BI or Metrics Store, as referred to here, is the hub for all metrics. Imagine it as the central repository of critical business logic, code, or metadata, implemented once and following the DRY principles.

Consider it as a repository of essential business logic, adhering to the DRY principles. This could be complex calculated measures in Apache Druid, for instance, that you wish to query across various BI tools or for diverse users. Instead of duplicating these measures in each tool, they’re centrally stored in the Analytics API, facilitated by the metrics store.

The communication within the Analytics API involves returning queries from the Query Engine via GraphQL, managing metadata store and data API configurations, and handling requests from the orchestration layer.

Currently, many rely on the popular Jinja and SQL templating solution, integrating it into the chosen BI tool. However, the rise of metrics stores is gaining momentum, primarily due to the complexities and limitations of re-using Jinja SQL logic across different database engines.

Metrics Stores, as integral parts of Analytics APIs, are crucial for analytics queries. The emerging role of Analytics Engineers, primarily tasked with defining and transforming these metrics, is becoming increasingly significant.

Moreover, the integration of additional services, akin to the Confluent Schema Registry (Schema Registry) for Kafka, could offer functionalities like comparing metric definitions and data store schemas, thus enhancing the robustness and consistency of data pipelines and application logic.

The data realm is buzzing with the concepts of Metrics Store and Headless BI. Noteworthy developments in this area include tools and platforms like Cube.js, Metriql, Supergrain, Transform.co, malloy, and AirBnB’s Minerva API. The adoption of GraphQL in these platforms is also notable.

Closed Sourced examples include Veezoo, uMetric, FlexIt with dbt

Drew Banin’s keynote at dbt’s Coalesce conference, discussing “ The Metric System”, is a must-watch for insights into the evolution of measurement and metrics.

Visualization:

Some thoughts:

• Comparing with Kimball’s approach: Data Modeling - Kimball, 3NF, Star Schemas, Activity Schema, What’s Best?

Related articles:

• Metrics Layers Compared
• dbt: Write once, query anywhere with the dbt Semantic Layer
• dbt: RW Getting Started With the Dbt Semantic Layer Dbt Developer Blog

References: Supergrain
Created: 2021-11-04