fix: a mid-stream backfill overwrites newer changes - #89
Merged
Merged
Conversation
A table added mid-stream has its backfill staged into its log alongside its CDC, in no useful order: a change streamed before the backfill reached its row was staged ahead of the row's older snapshot copy, and applied in log order the old copy won. A row deleted during the backfill came back, and an update leaving a TOASTed column unchanged had no row to resolve it from. Such a table's snapshot rows are now applied first, then its changes: - `register_table_pending` creates a second cursor for the table (`<group>#snapshot`), marking that its snapshot rows go first. - Once the backfill is complete, the materializer folds the log's snapshot rows in bounded steps and commits them together — the table goes from empty to its snapshot at once — then marks them applied. - Its changes then apply in log order, skipping the snapshot rows and any change committed before the first snapshot's LSN (the snapshot already reflects those). `SNAPSHOT_XID_BASE` moves to pg2iceberg-core (re-exported from pg2iceberg-snapshot) so the materializer can tell snapshot rows apart. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR fixes
backfill_does_not_overwrite_newer_changes.A table added mid-stream has its backfill staged into its log alongside its CDC, in no useful order. A change streamed before the backfill reached its row was staged ahead of the row's older snapshot copy, and applied in log order the old copy won. A row deleted during the backfill came back. An update leaving a TOASTed column unchanged had no row to resolve that column from.
Such a table's snapshot rows are now applied first, then its changes:
register_table_pendingcreates a secondmat_cursorrow for the table,<group>#snapshot. It marks, durably, that the table's snapshot rows go first.