To Nha Notes | June 6, 2025, 4:55 p.m.
As the volume and complexity of data continue to grow, traditional file formats and table management systems have started to show their limitations. In response, a new generation of open table formats has emerged—designed to bring reliability, scalability, and real-time capabilities to data lakes. Among the most prominent of these are Delta Lake, Apache Hudi, and Apache Iceberg.
Let’s take a brief look at what each of these modern table formats offers, and who’s using them.
Created by: Databricks
Open-source license: Linux Foundation project
Commercial version available: Yes
Delta Lake was developed by Databricks to bring ACID transactions, schema enforcement, and time travel to data lakes. It’s a powerful solution for teams already working within the Spark ecosystem but has also seen growing compatibility with engines like Presto, Snowflake, and Redshift.
Delta Lake is widely adopted due to its robustness and the fact that it's tightly integrated with the Databricks platform, though its open-source version ensures flexibility and access for all.
Created by: Uber
Open-source license: Apache Software Foundation project
Commercial version available: No (community-driven)
Apache Hudi (short for Hadoop Upserts Deletes and Incrementals) brings incremental data processing, upserts, and record-level operations to data lakes. It was built to handle high-throughput pipelines at Uber and later donated to the Apache Software Foundation.
Hudi has since gained traction in production environments across a range of industries. Companies like Amazon Transportation Service, Walmart, Robinhood, and GE Aviation have published case studies or blogs highlighting its use.
Created by: Netflix
Open-source license: Apache Software Foundation project
Commercial version available: No (but widely supported)
Apache Iceberg is designed to work at petabyte scale, offering advanced table evolution, hidden partitioning, and strong support for SQL-based analytics. Initially developed by Netflix, it’s now maintained as a community-led Apache project.
Iceberg’s popularity has skyrocketed, with contributions and usage from companies like Apple, Adobe, Airbnb, Expedia, and Lyft. Its compatibility with engines like Spark, Trino, Flink, and Hive makes it one of the most versatile table formats available today.
While each of these table formats has its strengths, your choice should depend on your specific use case:
Need tight Spark integration and enterprise support? Consider Delta Lake.
Working with high-frequency updates and real-time ingestion? Apache Hudi may be a great fit.
Want a scalable, flexible format with strong community backing? Apache Iceberg could be your best bet.
The good news is, all three are open-source and widely supported by modern data tools—meaning your data lake doesn’t have to be a data swamp anymore.