Introducing DuckLake: A Simpler, SQL-First Lakehouse Format

To Nha Notes | June 2, 2025, 10:47 a.m.

DuckLake is a new open-source lakehouse format developed by the DuckDB team, aiming to simplify data lakehouse architectures by leveraging SQL databases for metadata management and storing data in open formats like Parquet.(LinkedIn)

🦆 What is DuckLake?

DuckLake reimagines the lakehouse model by centralizing metadata in a SQL database, eliminating the need for complex file-based systems. This approach allows for efficient metadata operations using standard SQL queries, enhancing reliability and performance. (LinkedIn, DuckDB)

🔑 Key Features

• SQL-Centric Metadata: All metadata, including schemas and snapshots, are managed within a SQL database, streamlining operations and reducing complexity.

• Open Format Storage: Data is stored in Parquet files, maintaining compatibility with existing data processing tools.

• Simplified Architecture: By using SQL databases for metadata, DuckLake avoids the proliferation of small metadata files common in other lakehouse formats.

• Flexible Deployment: Supports various SQL databases like PostgreSQL, MySQL, SQLite, and DuckDB, allowing for versatile deployment scenarios. (Medium, Hacker News, DuckLake)

🚀 Getting Started

To use DuckLake with DuckDB:(DuckLake)

INSTALL ducklake;
ATTACH 'ducklake:metadata.ducklake' AS my_ducklake (DATA_PATH 'file_path/');
USE my_ducklake;
CREATE TABLE my_table(id INTEGER, val VARCHAR);
INSERT INTO my_table VALUES (1, 'Hello'), (2, 'World');
SELECT * FROM my_table;

This setup stores metadata in a DuckDB database file and data in Parquet files within the specified directory. (GitHub)

🧠 Why It Matters

DuckLake offers a streamlined, SQL-based approach to managing lakehouse architectures, reducing complexity and improving performance. Its open-source nature and compatibility with existing tools make it an attractive option for data engineers seeking efficient and flexible data lakehouse solutions.(Threads)

For more information, visit the official DuckLake page or explore the GitHub repository.(DuckDB)

📚 References

1. DuckDB Team. (2025, May 27). DuckLake: SQL as a Lakehouse Format. Retrieved from https://duckdb.org/2025/05/27/ducklake.html

2. https://ducklake.select/

3. https://github.com/duckdb/ducklake

4. https://www.reddit.com/r/dataengineering/comments/1kwnx2o/ducklake_a_new_datalake_format_from_duckdb/

5. https://medium.com/@shahsoumil519/learn-how-to-use-ducklake-with-duckdb-to-build-iceberg-tables-on-s3-274ca0495261

6. https://www.definite.app/blog/iceberg-query-engine

7. https://www.definite.app/blog/cloud-iceberg-duckdb-aws