RisingWave vs Apache Flink for Iceberg Ingestion: Why RisingWave Wins

Introduction

Most teams adopting Apache Iceberg treat "writing to Iceberg" as a connector problem: pick a streaming engine, configure a sink, and move on. In practice, a production Iceberg ingestion pipeline has to solve a much larger set of problems:

Exactly-once commits that survive task restarts and coordinator failover.
Commit cadence that is fast enough for freshness but not so fast that it poisons the table with small files.
Scheduled compaction, snapshot expiration, and orphan file cleanup.
Upsert and delete semantics for CDC workloads.
A query path for the data that is now sitting in Iceberg.

When these responsibilities are spread across a streaming engine, a sink library, a batch maintenance job, and an external query engine, data correctness becomes a cross-component contract. Every failure mode has to be reasoned about across system boundaries, and every component has its own operational surface.

RisingWave takes a different approach. It treats Iceberg ingestion as a native capability of a streaming database, not as a sink bolted onto a compute job. This article walks through the technical differences a data engineer should care about when choosing between a DIY pipeline built on Apache Flink and a managed, system-level path built on RisingWave.

What a Production Iceberg Ingestion Pipeline Actually Requires

Before comparing engines, it is worth being precise about the scope. A production pipeline needs at minimum:

Exactly-once writes with deterministic recovery after failure.
Bounded commit frequency decoupled from internal checkpoint frequency.
Compaction for small files, delete files, and manifest churn.
Snapshot expiration and orphan file cleanup to prevent unbounded storage growth.
CDC-friendly write semantics including primary keys, upserts, and deletes.
Read isolation so in-flight writes do not degrade or invalidate queries.
A query path for the Iceberg data, either for analytics or for serving.

In a Flink-based architecture, items 1 through 7 are delivered by a combination of a compute job, the Iceberg sink library, a separate batch maintenance pipeline (typically Spark), and an external query engine such as Trino or Spark SQL. Users are responsible for configuring each piece and for making them agree on state.

In RisingWave, items 1 through 7 are system capabilities. The rest of this article explains how.

1. Exactly-Once as a System Guarantee

RisingWave enables exactly-once Iceberg ingestion by default. The sink option is_exactly_once defaults to true, and the engine wires a two-phase commit coordinator behind every Iceberg sink:

The writer closes data and delete files when a checkpoint barrier arrives and forwards the resulting Iceberg metadata to the coordinator.
The coordinator persists the target Iceberg snapshot_id into the meta store's exactly_once_iceberg_sink_metadata table, keyed by sink id and epoch.
The coordinator then runs the Iceberg transaction. If that snapshot_id already exists in the Iceberg table history, the commit is skipped: this is how the system handles recovery without producing duplicates.
On restart, the coordinator replays from the last recorded epoch and uses the meta store to decide whether to re-emit or skip.

What this means in operational terms: a data engineer does not need to monitor "checkpoint advanced but Iceberg commit failed" as a separate failure mode, and does not need to build a dashboard or on-call runbook around metrics like elapsedSecondsSinceLastSuccessfulCommit. Commit correctness is closed inside the engine's state machine, not delegated to the user.

This matters precisely because in a typical Flink-based pipeline, commit responsibility sits at a boundary: Flink's checkpoint completes, and a separate callback triggers the Iceberg commit. A success on one side does not imply success on the other, which is why external monitoring is routinely recommended. RisingWave removes that boundary.

2. Commit Cadence as a Scheduling Policy

An Iceberg commit is a metadata transaction. Each commit produces a new snapshot, new manifests, and a new metadata.json. Commit too often and the table accumulates thousands of small snapshots and small files; read performance degrades and storage costs grow. Commit too rarely and freshness suffers.

RisingWave decouples internal checkpoint frequency from Iceberg commit frequency with a single sink parameter:

CREATE SINK orders_iceberg FROM orders
WITH (
    connector = 'iceberg',
    type = 'upsert',
    primary_key = 'order_id',
    warehouse.path = 's3://lake/warehouse',
    database.name = 'ods',
    table.name = 'orders',
    commit_checkpoint_interval = 60
);

The default value of commit_checkpoint_interval is 60. RisingWave can still advance computation at sub-second checkpoint granularity, but Iceberg commits happen once every 60 checkpoints. The system ships with a default that balances freshness and file hygiene, so most teams ship to production without tuning.

In pipelines where Iceberg commits are not a first-class scheduling primitive, users end up tuning Flink checkpoint interval, sink rolling policy, and a separate compaction schedule in parallel, hoping the three line up. RisingWave collapses all three concerns into one parameter, backed by the engine.

3. Compaction and Garbage Collection Are Built In

Iceberg table health depends on continuous maintenance: compacting small data and delete files, expiring old snapshots, and removing orphan files left by failed jobs. These tasks are not optional. Skip them and query latency grows, metadata traversal slows, and storage bills climb.

RisingWave integrates maintenance into the write path:

After a successful commit, the sink publishes compaction context (target table, snapshot, mergeable range) to the meta service.
The meta service maintains a compaction track and triggers work by snapshot count or by elapsed time.
A dedicated compactor pool receives scheduled tasks and executes them against the Iceberg table.
An independent garbage collection loop handles snapshot expiration and orphan file cleanup.

Data engineers who have run Iceberg at scale know the alternative: spin up a Spark cluster for RewriteDataFiles, another schedule for ExpireSnapshots, and often a third path for orphan cleanup, all orchestrated by Airflow or Dagster, none of which share state with the writer. Any drift between write rate and maintenance rate becomes a production incident waiting to happen. RisingWave removes that drift by making the writer and the maintainer part of the same system.

4. Upsert, Merge-on-Read, and Copy-on-Write as Table Contracts

Most real ingestion workloads come from databases, not append-only event streams. That means the pipeline has to handle primary keys, updates, and deletes end to end. RisingWave treats these as table-level contracts:

CREATE SINK ... FORMAT UPSERT requires an explicit primary_key. Append-only sinks forbid declaring one. The boundary between the two modes is enforced by the engine, not left to convention.
Append-only sinks are pinned to merge-on-read, because copy-on-write has no meaning for append-only data.
Upsert sinks with write_mode = 'copy-on-write' stage writes to an Iceberg ingestion branch. Readers on the main branch see a consistent, merged result, and the write path does not contend with live queries.
The table engine path, CREATE TABLE ... ENGINE = iceberg, extends this further. It supports write_mode selection and a VACUUM FULL statement that publishes compacted results to the main branch. These semantics are part of RisingWave's regression test suite, not an advanced user pattern.

The outcome for a data engineer is that upsert, MOR, and COW are described at the table level, once, and enforced by the system. You are declaring how the table should be maintained, not stitching together a sink configuration.

5. One System for Ingestion and Queries

Iceberg is a table format, not a query engine. In a pipeline where the streaming engine only writes to Iceberg, serving that data still requires a separate query system: Trino, Spark SQL, Dremio, or equivalent. That adds another cluster to operate, another SQL dialect to maintain, and another hop of latency between fresh data and the business.

RisingWave is PostgreSQL wire protocol compatible and serves queries out of the same materialized view system that feeds the Iceberg sink. A single SQL definition expresses ingestion, transformation, lake output, and a query endpoint:

CREATE TABLE orders (
    order_id BIGINT PRIMARY KEY,
    user_id BIGINT,
    amount DECIMAL,
    created_at TIMESTAMPTZ
) WITH (
    connector = 'kafka',
    topic = 'orders',
    properties.bootstrap.server = 'kafka:9092'
) FORMAT DEBEZIUM ENCODE JSON;

CREATE MATERIALIZED VIEW revenue_by_user AS
SELECT user_id, sum(amount) AS revenue
FROM orders
GROUP BY user_id;

CREATE SINK revenue_iceberg FROM revenue_by_user
WITH (
    connector = 'iceberg',
    type = 'upsert',
    primary_key = 'user_id',
    warehouse.path = 's3://lake/warehouse',
    database.name = 'analytics',
    table.name = 'revenue_by_user',
    commit_checkpoint_interval = 60
);

This single definition delivers four guarantees in one system:

Debezium CDC is ingested with upsert and delete semantics preserved.
revenue_by_user is an incrementally maintained materialized view. Every input change updates the result.
The same materialized view drives an exactly-once Iceberg sink with system-scheduled commits.
Applications can query revenue_by_user directly over the Postgres protocol. Fresh results are available before the next Iceberg commit.

RisingWave also supports Iceberg as a source. The source implementation scans the current snapshot first, then continuously reads new snapshots as they appear. A RisingWave pipeline can therefore consume data that other systems write to Iceberg and produce data for Iceberg in the same cluster, without external orchestration.

6. Architecture Comparison for a Data Engineer

Consider a realistic target: CDC from a Postgres database into an Iceberg lakehouse, with aggregations, upserts, and a query interface for the business.

A Flink-based pipeline typically requires:

A CDC connector process.
A Flink cluster running the compute job, with its own state backend and checkpoint storage.
The Iceberg sink library with exactly-once configured and externally monitored.
A scheduled batch job for file compaction, usually on Spark.
A scheduled batch job for snapshot expiration and orphan file cleanup.
An external query engine such as Trino or Spark SQL for serving.
An orchestrator (Airflow, Dagster) to keep the maintenance jobs in step with the writer.

Each component brings its own upgrade cycle, its own failure modes, and its own security surface. The contract between them is implicit, expressed in metadata and file timestamps. Data engineers spend more time on integration than on data.

A RisingWave pipeline for the same target is a single cluster that ingests CDC, runs SQL transformations, writes Iceberg with exactly-once and scheduled commits, runs compaction and GC internally, and serves queries over the Postgres protocol. The operational footprint is one system.

7. Capability Matrix

Capability	How RisingWave delivers it
Exactly-once commits	Coordinator with two-phase commit; `exactly_once_iceberg_sink_metadata` records snapshot id for idempotent recovery; closed inside the engine.
Commit cadence	`commit_checkpoint_interval` decouples checkpoint frequency from commit frequency. Default 60 balances freshness and file hygiene.
Compaction	Sink publishes compaction context to meta after commit; dedicated compactor pool executes scheduled tasks.
Snapshot GC	Independent GC loop handles snapshot expiration and orphan file cleanup.
Upsert semantics	`FORMAT UPSERT` requires a primary key; append-only sinks forbid one. Enforced by the engine.
Copy-on-write	Upsert with `write_mode = 'copy-on-write'` stages writes to an Iceberg ingestion branch; readers on the main branch see consistent results.
Iceberg source	Current snapshot scan followed by continuous incremental reads of new snapshots.
Query serving	Postgres-compatible wire protocol; materialized views queryable directly, no external engine required.

Conclusion

When data engineers evaluate an Iceberg ingestion engine, the important question is not "can it write Parquet to S3 and call Iceberg commit." It is whether the engine makes correctness, freshness, file hygiene, CDC semantics, and query serving system guarantees, or whether it offloads those concerns to the operator.

RisingWave makes them system guarantees:

Exactly-once is closed inside the engine, not monitored from outside.
Commit cadence is a first-class scheduling parameter with a validated default.
Compaction and GC run as native engine capabilities, sharing state with the writer.
Upsert, MOR, and COW are enforced as table-level contracts.
The same system that writes to Iceberg also serves queries over the Postgres protocol.

For teams building a real-time lakehouse in 2026, these are the properties that determine whether the pipeline is reliable on day 400 of operation, not just day 1. RisingWave is the engine that treats Iceberg ingestion as a production database feature. That is why it is the right choice.