Data engineer job description and skills

Last updated Feb 9, 2024

If English is the language of business, SQL is the language of data and Python is the language of engineering. While technology disappears often, SQL is still here. This means you need a reliable understanding of:

• SQL to a high level of complexity
  • e.g. using window and aggregate functions, changing the structure of a database with DDL or changing the data with DML, fine-tuning or reading database execution plans, how indices
• Data modeling techniques: ERDs and dimensional modeling
• Solid ETL understanding
• Architectural projections: needs to have a high-level understanding of most of the tools, platforms, libraries, and other resources at its disposal
• able to connect the dots from source to destination with any programming language that does the job best. (Probably Python at the moment)

Stitchdata.com anticipated that as company size increased, so would the focus on scaling-related skills. However, that’s not the story the data told. Instead, data engineers at larger companies tend to be more focused on enterprise skills like ETL, BI, and data warehousing, whereas data engineers at smaller companies focus more on core technologies:

Skill differences of data engineers across company sizes

References: Python ETL