🧠 Second Brain


Search IconIcon to open search

The role of a data engineer

Last updated Feb 9, 2024

In order to get high-quality and frequently updated data sets, it is important to distinguish between data pipelines that are done and cleaned by data engineers and all the others that are mostly exploratory. We at Airbus use a folder that is called “cleaned” and all data sets produced there are constantly updated, documented, and of the highest quality. Based on these data sets you create your own. We use the data lake solution Palantir Foundry (brand name of Airbus: Skywise) which provides you with a map where you see the data lineage easily. Documentation and metadata to each data set are crucial as otherwise, you lose the overview of your data, which is also one main task of a data engineer.

# Services that a data engineer provides

Another important task or service that a data engineer provides is automation which data scientists or data analysts do manually. A good overview of what task this includes are provided by Maxime Beauchemin, the founder of Apache Airflow, a tool that helps a data engineer to lift the majority of tasks mentioned:

While the nature of the workflows that can be automated differs depending on the environment, the need to automate them is common across the board. By Maxime Beauchemin

References: Data Engineering The role of a data engineer When is a data engineer needed