🧠 Second Brain
Search
ETL is changing
Furthermore, the way we do ETL is changing as well, as Maxime Beauchemin, data engineer at Airbnb quotes:
_Product know-how on platforms like Informatica, IBM Datastage, Cognos, AbInitio or Microsoft SSIS isn’t common amongst modern data engineers, and being replaced by more generic software engineering skills along with understanding of programmatic or configuration driven platforms like Airflow, Oozie, Azkabhan or Luigi. It’s also fairly common for engineers to develop and manage their own job orchestrator/scheduler.
He is also saying that code is the best abstraction there is for software rather than using drag-and-drop tools (ETL-tools). Most important what I see as well, is that the transformation logic is of a higher need and shouldn’t be locked away exclusively for BI developers.
References: ETL Business Intelligence Engineer vs Data Engineer