š§ Second Brain
Search
No-Code / Less-Code vs Code
No-Code is great for lowering the technical barriers and enabling more users to do data engineering. But these tools come with significant drawbacks:
- You are building knowledge in a proprietary tool rather than something generic and easily transferable (e.g., coding in Python)
- It’s powerful for a simple pipeline, but when it starts to be complex, it’s a nightmare to extend and maintain
- It’s not easy/impossible to follow best practices of software engineering like testing or versioning
- Licensing is usually pretty expensive.
- source RW No-Code vs Code - Databricks
Podcast
A marvelous podcast about the Modern Data Stack, how we solved the “4 Vās of #bigdata” and where we are now at the problem of #bigcomplexity. As well that we recycle the history with #lowcode / #nocode similarly to what we did with drag-and-drop #etl tools that didn’t remain because of the famous quote of Maxime Beauchemin: “#code is the best abstraction there is for #software”. Featured by Nick Schrock and Scott Breitenother.
https://databand.ai/mad-data-podcast/hello-big-complexity-is-your-modern-data-stack-ready/
A comment in regards of Closed-Source Data Platforms:
I replied in this comment on LinkedIn: I’d disagree that tools like Keboola or others are the solutions as you create all of the business logic within propitiatory (even with drag-and-drop) solutions again. Sure, you get all the value advantages you mention, which I fully agree with, but in the long run, it will be the same pain to migrate to anything else if you outgrow the solutions as with the SAPs and Oracles.
My hope, and the best solution for the modern data stack, is to limit the use of tools (don’t go above ten tools) and orchestrate it with Dagster or similar declarative and code-first tools. Your mentioned meta layer will be open source, and integration with even close source data solutions will be easier. But don’t put that automation inside a proprietary format. The key is an open declarative approach, let’s say a YAML or Python definition, then you are safe. If these tools support this, I’d approve of such solutions.
But having an integrated data platform instead of a scattered modern data stack is the trend I see as well :)
References:
Mehdi OUAZZA on LinkedIn: Databricks Acquires Low-code/No-code Company to Expand its Lakehouse,
Databricks Acquires Low-code/No-code Company to Expand its Lakehouse Platform to Citizen Data Scientists - Databricks, bamboolib, Medium Code
Created 2021-10-21