Delta Lake

Delta Lake is an Open Table Format built on top of Cloud Object Storage (like S3, GCS, ADLS) or distributed file systems (like HDFS). It enhances data lakes by bringing ACID transactions, Time Travel (data versioning), scalable metadata handling, and unified batch/streaming capabilities to data stored typically in Parquet format.

Developed initially by Databricks and now an open-source project under the Linux Foundation, Delta Lake aims to provide the reliability and performance typically associated with data warehouses directly on the vast, inexpensive storage of data lakes, forming the foundation of a Data Lakehouse architecture.

Key Features

• ACID Transactions: Ensures data integrity by making operations atomic, consistent, isolated, and durable. This prevents corrupted data resulting from concurrent writes or failed jobs. Transactions are managed through an ordered Transaction Log (_delta_log directory) stored alongside the data files.
• Scalable Metadata Handling: The transaction log serves as a central source of truth for table metadata, efficiently handling metadata for potentially billions of files, overcoming limitations of traditional directory listings on object stores.
• Time Travel (Data Versioning): Every operation on a Delta table creates a new version. Users can query previous versions of the table by timestamp or version number, enabling reproducibility, auditing, and easy rollbacks.
• Schema Evolution: Allows users to safely evolve the table schema (add/remove/modify columns) without rewriting the entire dataset. Schema enforcement prevents accidental insertion of data with mismatched schemas.
• Unified Batch and Streaming: Delta tables can serve as both a batch source/sink and a streaming source/sink, simplifying architectures by eliminating the need for separate systems (e.g., a Lambda architecture).
• OPTIMIZE Command: Includes operations like compaction (rewriting small files into larger ones) and Z-Ordering (colocating related information in the same set of files) to improve query performance.
• VACUUM Command: Cleans up old, unreferenced data files that are no longer needed by any table version within a specified retention period, helping manage storage costs.
• Open Source: Ensures vendor neutrality and fosters a broad community and ecosystem.

Common Use Cases

• Building reliable Data Lakehouses for BI and SQL analytics.
• Creating robust batch ETL/ELT pipelines on data lakes.
• Implementing reliable streaming data sinks into the lakehouse.
• Simplifying CDC (Change Data Capture) ingestion pipelines.
• Serving as a unified source for both data science and BI workloads.

Delta Lake vs. Other Formats

Delta Lake is often compared to other open table formats:

• Apache Iceberg: Both offer ACID transactions, time travel, and schema evolution. Key differences lie in their metadata structure (Iceberg uses hierarchical metadata files - manifests/manifest lists; Delta uses a linear transaction log), concurrency control mechanisms, and specific feature implementations (e.g., partitioning evolution, hidden partitioning specifics). Iceberg is often seen as potentially more scalable for extremely large numbers of files due to its metadata pruning capabilities, while Delta Lake's log-based approach can be simpler for certain operations.
• Apache Hudi: Hudi also provides ACID transactions and upsert/delete capabilities, offering Copy-on-Write and Merge-on-Read table types with different write/read performance trade-offs. Hudi's timeline concept is similar to Delta's transaction log but with different implementation details and features focused on incremental processing and write optimization.

The choice between them often depends on specific workload patterns, existing ecosystem integrations (Delta Lake has strong integration with Databricks/Spark), and feature preferences.

Delta Lake and RisingWave

RisingWave interacts with Delta Lake primarily as a potential sink target within a Data Lakehouse architecture:

• Sink Connector: RisingWave offers a sink connector that allows writing processed data streams into Delta Lake tables. This enables users to populate or update Delta Lake tables in near real-time based on computations performed in RisingWave.
• Streaming Lakehouse Pattern: Similar to the Iceberg pattern, using RisingWave to sink data into Delta Lake allows organizations to build a Streaming Lakehouse, providing fresh data in the lakehouse managed with Delta Lake's reliability features.
• (Future) Source: While sinking is the primary interaction, sourcing data from Delta Lake tables into RisingWave could be a potential future capability, allowing RisingWave to process historical lakehouse data or join streams against Delta tables.

The Delta Lake sink provides an alternative to the Iceberg sink for users who have standardized on Delta Lake as their open table format within their data lakehouse.

Related Glossary Terms

• Open Table Format
• Apache Iceberg / Apache Hudi (Alternatives)
• Data Lakehouse
• Streaming Lakehouse
• ACID Transactions
• Time Travel (in Lakehouse)
• Schema Evolution (in Lakehouse)
• Data Lake Compaction ('OPTIMIZE' command)
• Cloud Object Storage
• Parquet (File Format)
• Transaction Log (Concept)
• RisingWave Sink