Open-Source Data Engineering Projects

Last updated May 22, 2023

Mine:

• GitHub - sspaeti-com/practical-data-engineering: Real estate dagster pipeline (accompanied blog article: Building a Data Engineering Project in 20 Minutes)
• GitHub - airbytehq/open-data-stack: Open Data Stack Projects: Examples of End to End Data Engineering Projects

Other projects:

• GitHub - kotekaman/elt-with-dagster
• Data Engineering Project for Beginners - Batch edition
• HashtagCashtag
• Github Data Analytics
• Crawling For Inflation
• Mehdi Ouazza: GitHub - mehd-io/de-roadmap-projects-funnel: Data Engineer Roadmaps as Projects Funnel
• ngods (new generation open-source data stack): GitHub - zsvoboda/ngods-stocks: New Generation Opensource Data Stack Demo
  • Aricle: Iceberg + Spark + Trino + Dagster: modern, open-source data stack demo | by ZD | Jul, 2022 | Dev Genius
• GitHub - jacob1421/RustCheatersDataPipeline: Data pipeline that scrapes Rust cheater Steam profiles
• duckdb - dbt- meltano -superset: GitHub - matsonj/nba-monte-carlo: Monte Carlo simulation of the NBA season, leveraging meltano, dbt, duckdb and superset ^dcb4c6
  • blog article: DuckDB - Modern Data Stack in a Box with DuckDB
• GitHub - davidgasquez/open-source-data-stack: 🧱 Tinkering around with a bunch of open source data tools.
• Build a poor man’s data lake from scratch with DuckDB | Dagster Blog
• GitHub - stkbailey/dagster-jaffle-shop: Repo for orienting dbt users to the Dagster asset framework
• Airbyte Monitoring with dbt and Metabase - Part I | Airbyte
  • GitHub: open-data-stack
• 📈 Stock Market Real-Time Data Analysis Using Kafka | End-To-End Data Engineering Project - YouTube
  • GitHub - darshilparmar/stock-market-kafka-data-engineering-project
• https://www.linkedin.com/feed/update/activity:7012604955215740928
• Building blocks for Apache Airflow: Astronomer Registry
• GitHub - damklis/DataEngineeringProject: Example end to end data engineering project.
• GitHub - dagster-io/fake-star-detector
• GitHub - cogitovirus/gcs-modern-data-stack: Demo data stack using Dagster + Airbyte + dbt
• Data Engineering Practice Problems
• GitHub - mddan/news_etl: News ETL CAPSTONE project for Data Engineering Bootcamp
• Shopify ETL: GitHub - VincentFaust/willowsspoon

GitHub Lists:

• sspaeti’s list / DE projects · GitHub
• GitHub - gunnarmorling/awesome-opensource-data-engineering: An Awesome List of Open-Source Data Engineering Projects

Articles:

• Big Data Pipeline Recipe

Youtube:

• 5 Data Engineering Projects Ideas To Put On Your Resume - YouTube
• How To Start A Data Engineering Project - With Data Engineering Project Ideas - YouTube

Bootcamps:

See also Find good Data Sets or Sources and Open-Source ML Projects

Origin:
References:
Tags: #🗃/🌻 #publish
Created 2022-08-02