Data Lake File Formats

Last updated Sep 4, 2024

At the core of a Data Lake is the Storage Layer, which predominantly houses files in open-source formats.

These data lake file formats are essentially the cloud’s version of CSVs but with enhanced capabilities. They are more column-oriented, facilitating efficient compression of large files and incorporating additional features. The leading formats in this space are Apache Parquet, Apache Avro, and Apache Arrow. They constitute the physical storage, with files distributed across various buckets within your storage layer.

The utility of data lake file formats extends beyond mere storage. They play a crucial role in data sharing and exchange between different systems and processing frameworks. Key attributes of these formats include their ability to be split and their support for schema evolution, making them versatile across various teams and programming languages.

• Apache Parquet
• Apache Avro
• Apache Arrow
• ORC
• CSV
• newer formats, try to improve parquet:
  • Nimble
  • Lance
• DIDI File Format

Great overview by Thalia Barrera on File formats - Complete Guide to Data Lakes and Lakehouses Video Tutorial:

Selecting the appropriate data lake file format hinges on specific requirements. Apache Parquet is gaining traction for its efficiency. Avro stands out with its sophisticated schema description language and support for Schema Evolution. However, an interesting development to consider is the relationship between data lake table formats and their underlying file formats at a higher abstraction layer.

Comparison of Data Lake File Formats (Inspired by Nexla: Introduction to Big Data Formats)

📝 Note: The importance of Schema Evolution is somewhat mitigated, as data lake table formats (discussed in the next chapter) also provide support for this feature.

Star history comparing the different file formats.

Find more on Data Lake / Lakehouse Guide: Powered by Data Lake Table Formats (Delta Lake, Iceberg, Hudi) | ssp.sh.

Origin: Storage Layer
References: Big Data Pipeline Recipe - Full Article
Created 2022-08-15