Data Integration

Last updated Jun 26, 2025 by Simon Späti

Data integration is a crucial process in the world of data management, where we bring together data from varied source systems to create a cohesive, unified view. This integration can happen in multiple ways, such as through manual effort, data virtualization, application integration, or by migrating data from numerous sources into a singular, integrated destination. We delve into these methods of data integration in the discussion below.

For a deeper dive, see Data Integration Iceberg and explore more in the Data Integration Guide: Techniques, Technologies, and Tools | Airbyte.

Data Integration vs. Data Ingestion

Often, the lines between data integration and Data Ingestion seem blurred, with the differences appearing minimal. However, a subtle distinction lies in their scopes. Data ingestion is a broader concept, concerned with the movement of data from sources to destinations. On the other hand, data integration is more nuanced, focusing particularly on the consolidation of data within platforms like Data Warehouse, Data Lake, or other data platforms. For example, Apache Druid ingests data to Druid, not integrates. See Data Ingestion with Druid.

# Tools for Data Integration

High-level I would divide between:

CLI-first tools
Platforms: that come with components like webserver, scheduler, database, etc.

# Open-Source Tools

CLI first:

dlt: Python-based data integration library designed from the ground up to improve upon Airbyte/Meltano. Perfect fit as a library rather than platform, with well-thought-through architecture and comprehensive connector ecosystem
Sling: Similar to dlt, recommended by Dagster team. Works well integrated with Dagster orchestrator, supports incremental loading with custom SQL
Steampipe: SQL-based, very lean tool but lacks advanced features like incremental loading, full load functionality, and UI. Lower customization for non-existing connectors compared to dlt
ConnectorX: Fast connector for data transfer
ingestr: Simple one-to-one copy tool. For more complex scenarios, dlt is recommended as better alternative

UI and Code-First:

Airbyte: Easiest to use, UI-first approach but heavy runtime. Also supports Python approach through PyAirbyte for code-first users
Apache Nifi: Drag-and-drop UI-based workflow orchestrator, similar to Kestra but less automatable since Kestra generates YAML on each change
Meltano: Similar to Dagster but not recommended. Built on top of singer.io (mostly old taps). Pivoted to Open Source Data Ops OS after Airbyte’s success

Data Orchestrators that support data integration:

Apache Airflow: Most well-known orchestrator from Apache ecosystem with largest community support. However, it’s older with many limitations that newer tools like Dagster and Prefect have addressed
Dagster: Code-first, extensive with data engineering best practices. Python-based orchestrator requiring Python knowledge. Takes time to learn but works excellently with custom connectors integrated into the orchestrator
Kestra (orchestrator, but has plug and play plugins for integration): UI and Code-first (YAML-based) orchestrator. Easy to start, supports both simple and complex workloads. UI makes it accessible to anyone, while YAML format enables automation of pipelines

Specialized Open-Source

CloudQuery: Open-source cloud asset inventory tool
Vector.dev: Run by DataDog, data observability pipeline
Peer DB: Specialized for Postgres synchronization

# Commercial/Enterprise Tools

Cloud-First Platforms

Fivetran: Cloud-first, expensive but UI-first approach. Enterprise-grade data integration platform
Portable: Cheaper alternative to Fivetran, focuses on long-tail connectors to avoid direct competition with larger companies. Complements mainstream solutions with specialized one-off connectors
StitchData: Poor reputation for connector quality and customer service with uncertain future prospects. Low-cost approach with pricing based on total rows updated
HevoData: Enterprise data integration platform
Dataddo: Cloud-based data integration platform

Enterprise-Focused

Matillion: Self-hosted, enterprise-focused solution. Not considered direct alternative to cloud offerings due to different target company profiles
Estuary: Focuses on streaming data connectors with real-time and batch data optimization for speed, cost, and efficiency

AI-Enhanced