Data Integration CLI tools
Data Integration that supports CLI use cases.
# Tools
- dlt: Python-based data integration library designed from the ground up to improve upon Airbyte/Meltano. Perfect fit as a library rather than platform, with well-thought-through architecture and comprehensive connector ecosystem
- Sling: Similar to dlt, recommended by Dagster team. Works well integrated with Dagster orchestrator, supports incremental loading with custom SQL
- Steampipe: SQL-based, very lean tool but lacks advanced features like incremental loading, full load functionality, and UI. Lower customization for non-existing connectors compared to dlt
- ConnectorX: Fast connector for data transfer
- ingestr: Simple one-to-one copy tool. For more complex scenarios, dlt is recommended as better alternative
- SDF (now dbt Fusion)
- Meltano (bought, see Data Engineering Acquisitions)