Data Fusion

Last updated Jan 22, 2023

DataFusion is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.

DataFusion supports both an SQL and a DataFrame API for building logical query plans as well as a query optimizer and execution engine capable of parallel execution against partitioned data sources (CSV and Parquet) using threads.

DataFusion also supports distributed query execution via the Ballista (Arrow) crate.

DataFusion can be used without modification as an embedded SQL engine or can be customized and used as a foundation for building new systems. Here are some examples of systems built using DataFusion:

• Specialized Analytical Database systems such as  CeresDB and more general spark like system such a Ballista (Arrow)
• New query language engines such as  prql-query(PRQL) and accelerators such as  VegaFusion
• Research platform for new Database Systems, such as  Flock
• SQL support to another library, such as  dask sql
• Streaming data platforms such as  Synnada
• Tools for reading / sorting / transcoding Parquet, CSV, AVRO, and JSON files such as  qv
• A faster Spark runtime replacement (blaze-rs)

By using DataFusion, the projects are freed to focus on their specific features, and avoid reimplementing general (but still necessary) features such as an expression representation, standard optimizations, execution plans, file format support, etc.

• High Performance: Leveraging Rust and Arrow’s memory model, DataFusion achieves very high performance
• Easy to Connect: Being part of the Apache Arrow ecosystem (Arrow, Apache Parquet and Flight), DataFusion works well with the rest of the big data ecosystem
• Easy to Embed: Allowing extension at almost any point in its design, DataFusion can be tailored for your specific usecase
• High Quality: Extensively tested, both by itself and with the rest of the Arrow ecosystem, DataFusion can be used as the foundation for production systems.

Here is a comparison with similar projects that may help understand when DataFusion might be be suitable and unsuitable for your needs:

• DuckDB is an open source, in process analytic database. Like DataFusion, it supports very fast execution, both from its custom file format and directly from parquet files. Unlike DataFusion, it is written in C/C++ and it is primarily used directly by users as a serverless database and query system rather than as a library for building such database systems.
• Polars: Polars is one of the fastest DataFrame libraries at the time of writing. Like DataFusion, it is also written in Rust and uses the Apache Arrow memory model, but unlike DataFusion it does not provide SQL nor as many extension points.
• Facebook Velox is an execution engine. Like DataFusion, Velox aims to provide a reusable foundation for building database-like systems. Unlike DataFusion, it is written in C/C++ and does not include a SQL frontend or planning /optimization framework.
• DataBend is a complete database system. Like DataFusion it is also written in Rust and utilizes the Apache Arrow memory model, but unlike DataFusion it targets end-users rather than developers of other database systems.

Origin:
References: ROAPI Apache Arrow Rust
Created: 2021-10-14