ACID Transactions

ACID is an acronym representing four key properties that guarantee database transactions are processed reliably: Atomicity, Consistency, Isolation, and Durability. Traditionally associated with relational databases (RDBMS), these properties ensure data integrity even during errors, power failures, or concurrent operations.

In modern data architectures, particularly the Data Lakehouse, technologies like Open Table Formats (Apache Iceberg, Delta Lake, Apache Hudi) aim to bring ACID guarantees to operations on large datasets stored in Cloud Object Storage.

The ACID Properties

1. Atomicity: Ensures that all operations within a transaction are treated as a single, indivisible unit. Either all operations complete successfully, or none of them do. If any part of the transaction fails, the entire transaction is rolled back, leaving the database in its previous state.
2. Consistency: Guarantees that a transaction brings the database from one valid state to another. Any data written must be valid according to all defined rules, including constraints, cascades, and triggers. Consistency ensures that illegal transactions are prevented.
3. Isolation: Ensures that concurrent transactions occur without interfering with each other. The results of concurrently executing transactions should be the same as if they were executed sequentially. This prevents issues like dirty reads, non-repeatable reads, and phantom reads. Isolation is typically achieved through locking mechanisms or multiversion concurrency control (MVCC).
4. Durability: Guarantees that once a transaction has been successfully committed, it will persist permanently, even in the event of system failures like crashes or power outages. Committed data is typically written to non-volatile storage.

ACID in Data Lakes / Lakehouses

Applying ACID principles to data lakes presents challenges because the underlying storage (like S3 or GCS) often lacks native transaction support. Open Table Formats solve this by implementing transactional capabilities on top of the object store:

• Metadata Management: They use metadata files (like transaction logs or manifests) stored alongside the data files to track valid data files comprising a table version.
• Atomic Commits: Committing a change involves atomically updating a pointer to the latest valid metadata file. This makes operations like adding or removing multiple data files appear as a single atomic action.
• Isolation: Techniques like optimistic concurrency control or snapshot isolation are used, allowing writers to proceed assuming no conflict, and then checking for conflicts before committing. Readers typically read a consistent snapshot of the data based on a specific metadata version.
• Durability: Relies on the durability guarantees of the underlying cloud object storage for both data and metadata files.

Relevance to RisingWave

While RisingWave itself, as a streaming database, ensures consistency and durability for its internal state through mechanisms like Checkpointing and its Hummock state store, the concept of ACID transactions becomes particularly relevant when RisingWave interacts with external systems, especially within a Streaming Lakehouse architecture:

• Sink Connectors (Iceberg/Delta): When RisingWave sinks data into an Apache Iceberg or Delta Lake table, the corresponding sink connector interacts with the table format's transactional mechanism. This ensures that the data written by RisingWave (often representing changes from a materialized view) is committed atomically and consistently to the lakehouse table, making the updates visible to other query engines in a reliable manner.
• Data Integrity: Leveraging ACID-compliant sinks ensures that the results of real-time processing in RisingWave are reliably integrated into the broader data lakehouse ecosystem without risking data corruption or inconsistency.

Related Glossary Terms

• Open Table Format (Apache Iceberg, Delta Lake, Apache Hudi)
• Data Lakehouse
• Consistency (State Management)
• Durability (State Management)
• Checkpointing
• Cloud Object Storage
• Transaction Log (Concept)
• Sink