Overview
Backfilling is the process of using existing historical data to build the state of a data system. In broader data engineering, it involves filling missing records, correcting stale or incorrect data, recovering from pipeline downtime, reprocessing data after a logic change, or initializing a new pipeline from data that already exists. Depending on the scope, a backfill may cover a few specific records, a full range of records, some partitions, or the entire existing historical dataset. That’s why safe backfills should have a clearly defined scope so that we can correctly build an up-to-date state without exposing users to inconsistent, duplicated, or partially updated results.
In RisingWave, backfilling is a fundamental part of creating materialized views (MVs) on top of existing data. When a materialized view is created, it first processes the existing historical data from its upstream relations, such as an upstream table or another MV, and builds the initial query state. After this historical phase is complete, the MV transitions to incremental streaming and continuously updates as new data arrives. When a new streaming job backfills from an existing RisingWave relation, RisingWave supports multiple backfilling strategies, including snapshot backfill, arrangement backfill, and no-shuffle backfill. These strategies apply specifically to backfilling from existing RisingWave relations.
Similarly, loading existing data from an external CDC source, such as a PostgreSQL CDC table, is also considered backfilling, but it is handled separately using the CDC backfill executor rather than one of the three backfilling strategies above. Moreover, RisingWave also supports backfilling optimizations and resource placements, including locality backfilling and serverless backfilling, to make the process more efficient and better isolated from live streaming workloads.
Backfilling in RisingWave
In RisingWave, backfilling is the initialization phase that occurs when an MV is created on top of upstream data that already exists in a RisingWave table, another MV, or an upstream source such as a PostgreSQL CDC table. For example, consider an upstream table in RisingWave that contains millions of historical customer orders from a PostgreSQL CDC table, and a new MV is created to calculate total sales per customer. RisingWave must first read those existing records from the orders table, apply the query, build the aggregation state, and materialize the existing results. Only after that initial state is complete can the materialized view continue with normal incremental processing as new data arrives from the orders table.
This process is important because a streaming object must have the complete historical state from the moment it goes live. Without backfilling, a newly created MV would include only future changes and ignore the historical data that existed before it was created. Backfilling therefore connects historical state with live streaming state. RisingWave coordinates this transition using consistent snapshots, barriers, checkpoints, fragment-level progress, a log store for storing new data, and continuously persisted streaming job state, so the system can process historical data while also preserving the changes that occur during initialization.
Backfilling Strategies
RisingWave supports different backfilling strategies for initializing materialized views such as snapshot backfill, arrangement backfill, and no-shuffle backfill (deprecated now).
Snapshot Backfill
Snapshot backfill reads the upstream table or MV at a specific snapshot epoch. Then, it scans this snapshot in chunks while continuing to poll and buffer upstream data in the log-store. After the snapshot scan is complete, RisingWave consumes the changes that occurred after the snapshot epoch from the upstream log-store until it catches up with the upstream. Once this catch-up is finished, it transitions to consuming the normal live stream. The snapshot therefore provides a committed view of the upstream table or MV at the selected snapshot epoch, while later upstream changes are handled separately during the backfill process.
The main advantage of this approach is that RisingWave can directly read the table’s or MV’s state at the snapshot epoch instead of replaying older changes that originally created that state. It then consumes the change-log data from log-store needed to catch up after the snapshot before switching to normal upstream processing. This separates the initial snapshot scan from the later incremental changes.
Snapshot backfill has been enabled by default in RisingWave since v2.8. During the backfill process, progress is checkpointed at barriers so that, after a failure, RisingWave can resume from the previously saved backfill progress instead of necessarily restarting the entire backfill from the beginning.
Arrangement Backfill
Rather than copying the whole upstream table locally, arrangement backfill reads committed state from shared storage and locally replicates the upstream’s in-flight changes.
The snapshot is scanned gradually across vnodes, which are virtual partitions of the table. For each vnode, RisingWave remembers the last primary-key position it has already scanned. In the diagram, for example, vnode 0 has scanned up to pk=7, while vnode 1 has scanned up to pk=4. RisingWave keeps these positions as the current progress for each vnode. New upstream chunks that arrive while the snapshot scan is still in progress are replicated into the local ReplicatedStateTable.
When a barrier arrives, RisingWave first flushes any remaining snapshot rows and updates the current scan position for each vnode. Buffered changes whose primary keys fall within the already-scanned prefix can be forwarded downstream. All buffered upstream changes are also written to the replicated state table so that later snapshot scans see the updated table state. RisingWave then saves the backfill progress for each vnode and continues the next snapshot scan from the saved primary-key position rather than starting again from the beginning.
This process repeats until all required snapshot data across the vnodes has been scanned. RisingWave then marks the backfill as complete and switches to forwarding the normal live upstream directly downstream. Arrangement backfill can also run on different compute nodes and be scaled independently from the upstream fragment, instead of being required to use the same parallelism as the upstream.
Arrangement backfill became the default in RisingWave v1.8, while snapshot backfill became the default starting with v2.8. The enable_arrangement_backfill configuration option is deprecated. If snapshot backfill is disabled using SET streaming_use_snapshot_backfill = false, or if the query involves temporal joins or plans that use a shared source, RisingWave uses arrangement backfill instead. Otherwise, snapshot backfill is used by default.
No-Shuffle Backfill
No-shuffle backfill was a backfill mechanism that relied on the backfill executor and the upstream relation being placed together on the same worker. Because both run on the same worker, RisingWave can read the latest upstream data directly, even before it is fully committed to storage.
During backfill, RisingWave read the upstream snapshot and tracked progress using the upstream relation’s primary-key position. Upstream messages arriving between two barriers were buffered. At the later barrier, those buffered changes were either forwarded or ignored depending on whether their primary keys fell within the portion of the snapshot that had already been processed. Once the tracked primary-key position reached the end of the upstream table or MV, the backfill was considered complete. No-shuffle backfill has been deprecated in RisingWave.
Locality Backfilling
Locality backfilling is a backfill optimization that helps preserve data locality across the backfill pipeline. RisingWave’s optimizer can insert LocalityProvider operators at different points in a streaming query plan where locality backfilling is useful. A LocalityProvider is an intermediate backfill operator that buffers incoming data in a state table using locality columns as a primary-key prefix and then provides snapshot reads in locality order. In simple terms, it helps keep related rows closer together while they are being backfilled. This can improve cache locality and reduce random remote I/O during backfilling, though it does not mean that all network shuffles are eliminated across the entire pipeline.
Each LocalityProvider performs a backfill process similar to arrangement backfill. It buffers incoming upstream changes while reading a locality-ordered snapshot from its state table. The state table is used to hold the buffered data that the LocalityProvider later scans in locality order. Backfill progress is tracked separately for each vnode. For each vnode, RisingWave records the current primary-key position reached by the snapshot scan so that it knows how far that portion of the backfill has progressed.
When a barrier arrives, the LocalityProvider processes the buffered upstream changes and writes them to its state table. It also persists the backfill progress for each vnode. If buffered upstream changes modified the state table, the snapshot stream is reconstructed so that subsequent snapshot reads can see the latest state. If no buffered upstream changes modified the state table, the existing snapshot stream can continue across the barrier. Once the locality-ordered snapshot scan is complete, the LocalityProvider finishes its backfill processing and forwards upstream messages directly.
A single query plan can contain multiple LocalityProvider operators. Once locality backfilling is enabled, the optimizer automatically inserts these operators into newly created query plans where needed to provide data locality during backfilling. A query plan that uses more than five LocalityProvider operators requires the premium feature.
Locality backfilling was added in RisingWave v2.7 and is disabled by default. It can be enabled for the session with:
SET enable_locality_backfill = true;
Locality backfilling improves execution efficiency, but it does not remove the need to manage resource pressure. If CPU utilization, cache misses, memory pressure, or remote I/O become high during a large backfill, it can be more efficient to break the query into several smaller materialized views and create them one by one.
Serverless Backfilling
Backfilling historical data can consume substantial CPU, memory, storage I/O, network bandwidth, and compaction capacity. By default, the backfill phase of creating a materialized view, sink, or index runs on the same compute nodes as regular streaming workloads. Large backfills can therefore compete with existing jobs for resources and increase latency, especially when a query contains large joins, many CTEs or subqueries, or a significant amount of historical state.
Serverless backfilling in RisingWave Cloud runs this historical initialization phase on dedicated temporary backfiller resources instead of the main streaming compute nodes. After a CREATE MATERIALIZED VIEW statement is validated and the streaming job is ready to be created, RisingWave Cloud provisions a separate resource group using the configured backfiller SKU and replica count. If validation fails, these resources are not provisioned.
The backfill runs in this temporary resource group without directly competing with existing streaming jobs for compute resources. Once the backfill task is complete, the job moves to the steady-state resource group of its parent database for normal streaming execution, and the temporary backfiller resources are removed automatically. If the cluster restarts during backfilling, the job resumes from its most recently completed checkpoint instead of starting again from the beginning.
Serverless backfilling has been available since RisingWave v2.8.0, is disabled by default, and is supported only in RisingWave Cloud. Large backfills can also increase write throughput and compaction pressure. Compaction resources must therefore be sized for the temporary backfill load rather than only for steady-state traffic. For this reason, Serverless Compaction is recommended so that compactor resources can scale automatically. Otherwise, the compactor should be scaled manually before running heavy backfill jobs to avoid compaction back pressure or write stalls.
Serverless backfilling can be enabled for subsequent materialized view, sink, and index creation statements in the current session:
SET enable_serverless_backfill = true;
The default value is false, and the setting applies only to new DDL operations issued after the command. It does not change already-running jobs.
It can also be enabled for an individual materialized view using the statement-level WITH option:
CREATE MATERIALIZED VIEW mv
WITH (cloud.serverless_backfill_enabled = true)
AS
SELECT ...;
Backfill progress can be monitored through:
SELECT * FROM rw_catalog.rw_ddl_progress;
Users can also use SHOW JOBS to inspect background DDL jobs and view the dedicated risingwave-backfill-* node series on the RisingWave Cloud Metrics page to monitor backfiller CPU and memory usage.
Best Practices for Backfilling in RisingWave
Backfilling can place significant pressure on compute, storage, upstream systems, and compaction resources. The following practices can help improve backfill performance and reduce its impact on live workloads.
Schedule large backfills carefully. Create large materialized views during lower-traffic periods when possible. Even with serverless backfilling, large historical scans can still affect shared storage, upstream sources, object-storage bandwidth, and compaction.
Use serverless backfilling for heavy jobs. In RisingWave Cloud, dedicated backfiller nodes isolate much of the CPU and memory pressure from the main streaming compute nodes, reducing the impact on live streaming workloads.
Monitor backfill progress and resource usage. Use system catalogs and diagnostic tools such as
rw_catalog.rw_fragment_backfill_progress,rw_catalog.rw_ddl_progress, andDESCRIBE FRAGMENTS. Also monitor CPU and memory usage, barrier latency, checkpoint duration, storage I/O, and compaction pressure.Use background DDL for long-running operations. When the client should not remain blocked during materialized view creation, enable background DDL:
SET BACKGROUND_DDL = true;
Control rate limits and parallelism. Lower backfill parallelism can reduce resource contention, while normal streaming parallelism can be increased after initialization is complete. Limiting concurrent streaming job creation can also prevent multiple large backfills from exhausting cluster resources at the same time.
Break complex queries into smaller materialized views. For queries with many CTEs, subqueries, joins, or aggregations, consider creating several smaller materialized views one by one. A single SQL statement with many stateful branches can behave like several expensive queries running simultaneously, increasing memory pressure, cache misses, CPU usage, and remote I/O.
Improve data locality. For materialized views built on other materialized views, align the upstream order key with the downstream grouping, join, or partition key where possible. This allows related rows to be read consecutively, improves cache efficiency, and reduces random remote I/O. Index selection and locality backfilling can further improve this process.
Control the backfill order for joins. For join-heavy MVs, use
backfill_orderwhen appropriate to backfill smaller dimension tables before a large fact table. This can reduce failed lookups, retractions, and update churn caused by processing fact-table rows before the required dimension state is available. However,backfill_orderis still in technical preview. It is supported for both materialized views and sinks, and the requested order is persisted and restored during Background DDL recovery.Avoid unnecessary full rebuilds. If an expensive MV is repeatedly dropped and recreated, RisingWave must rebuild its historical state each time. Keep stable and expensive state in materialized views, and place frequently changing logic in lighter downstream views when possible.
Backfilling should be treated as a controlled production workflow. Define its scope, understand downstream dependencies, monitor its progress, validate the resulting data, and ensure sufficient compaction capacity is available.
Conclusion
Backfilling bridges historical initialization and continuous streaming, allowing newly created materialized views to begin with a complete state and remain up to date as new data arrives. With snapshot and arrangement backfill, locality-aware execution, serverless resource isolation, rate limits, parallelism controls, progress monitoring, and checkpoint-based recovery, RisingWave makes backfilling more efficient and resilient. Rather than treating historical data processing as a simple rerun, RisingWave makes it a first-class part of stream processing that can be carefully optimized and managed for production workloads at scale.
