SQL

SQL, a domain-specific language, is pivotal in programming, especially for managing data in relational database management systems or stream processing in relational data stream systems.

It’s closely tied to Jinja Template within dbt. Beyond Business Intelligence tools, a growing number of data engineering tools are now exclusively integrating with SQL. This is further explored in Building an Analytics API with GraphQL - The Next Level of Data Engineering.

For additional insights, refer to Components of an Analytics API.

SQL is the language of data. SQL is a fundamental skill for doing any data work. There’s almost nothing you do without needing SQL. If you work with a REST API with no direct SQL interface, it’s still beneficial to know, as the REST service will most certainly perform a SQL query against the database based on your REST request.

If you understand how joins and filters work, you can add a key filter to the request or split up a request into two, as you know it might be a heavy join behind, and that way you avoid it.

On the other hand, Python is a data engineer’s tooling language. It’s used to glue the different steps of a data engineer together. It’s the ultimate toolkit language. Pulling data from a REST API or web, cleaning out some insufficient data, and storing it in Postgres. How would you do that in a safe, ordered fashion? Right, Python.

In the old days, we used stored procedures. If you live in one ecosystem of large vendors like SAP, Oracle, or Microsoft, you can add your glue code into the database code with PL/SQL or T-SQL. But if you work with any outside tool, like a REST API, the tooling around Python is so massive and extensive that you will most certainly make your life easier by using Python.

SQL was invented in the 1970s based on the Relational Data Model. It was initially known as the structured English query language (SEQUEL). The term was later shortened to SQL.

Oracle, formerly known as Relational Software, became the first vendor to offer a commercial SQL relational database management system.

“History of SQL”

SQL -> Data Mart -> Materialized View -> BI Report -> Traditional OLAP -> BI Dashboard -> Modern OLAP -> dbt tables -> One Big/Wide/Super Table -> Semantic Layer -> Natural Language Queries

Since then, SQL has come to dominate the landscape. (“Never bet against SQL,” I like to say.) Most users have moved to cloud SQL services (BigQuery, Snowflake) because convenience is more important than the flexibility and raw power of Apache Hadoop or Spark. SQL now handles workloads—streaming, data transformation, documents, geospatial, machine learning—previously handled by non-SQL systems. But there are two areas where SQL has not yet prevailed—Business Intelligence and data-intensive computing—and I want to investigate whether the solution is SQL or something else. Julian Hyde on Into the Wilderness

What is ANSI SQL?

• Nice new features: UNION ALL BY NAME

SQL is declarative.

SQL is the origin of data, and it has been extended over the years. We have geospatial capabilities, window functions, and other features throughout the year. Why not add analytics semantics, too?

See Extending SQL for analytics.

From parsing to compiling and executing. Possible with SDF and SQLGlot:

source

Julian Hyde in 2025-06-04 at data council More Than Query Future Directions of Query Languages, from SQL to Morel - YouTube:

From Matthias Broecheler on LinkedIn:

Everything gets pulled back in.

• MapReduce? Became Hive. Then SparkSQL.
• NoSQL? Morphed to “Not Only SQL” and then NewSQL.
• Graph? Led to pattern matching inside SQL.
• Vectors? SQL index type
• Streaming? Enter: Flink SQL.
  He shares more on The Gravitational Pull of SQL - by Matthias Broecheler.

• Recommended Reading: Efficient Data Processing in SQL by Joseph Machado
• Comprehensive Guide: Ultimate SQL Guide from Count-co
• Learning SQL as a detective: SQL Noir

Origin:
References:
Created 2022-08-08