SDSTOR-23112: fix snapshot shard state when sealed_lsn exceeds snapshot LSN cutoff#451
Merged
Merged
Conversation
…ot LSN cutoff
## Problem
During snapshot generation, PGBlobIterator builds a shard list from the
leader's in-memory PG state filtered by `upto_lsn` (the snapshot LSN
cutoff). Before this fix, a shard whose `create_lsn <= upto_lsn` was
always included with its current state — even if its `sealed_lsn` was
beyond `upto_lsn` (i.e., the seal happened *after* the snapshot point).
When a follower received such a snapshot it saw the shard as SEALED and
released the backing chunk. Log replay then tries to replay the
`seal_shard` entry for that LSN, but the chunk has already been
reclaimed or relocated, causing the commit to fail with stale
physical-chunk references.
## Root Cause
The race window is:
1. Leader creates shard S at LSN L1.
2. Leader creates shard S2 at LSN L2 (L2 > L1).
3. Leader seals shard S at LSN L3 (L3 > L2).
4. Snapshot is taken with `upto_lsn = L2`.
In step 4, S has `create_lsn = L1 <= L2` so it passes the LSN filter,
but `sealed_lsn = L3 > L2`. Old code included S with state SEALED.
The follower then applied the snapshot, treated S as SEALED, released
its chunk, and later failed when replaying the seal_shard log at L3.
## Fix
In `PGBlobIterator::PGBlobIterator`, after the `create_lsn` filter,
add a second check:
if (shard_info.state == SEALED && shard_info.sealed_lsn > upto_lsn) {
shard_info.state = OPEN;
shard_info.sealed_lsn = INT64_MAX;
}
This converts the shard's snapshot state back to OPEN so the follower
does not release the chunk prematurely. The follower will replay the
`seal_shard` log entry when log replay catches up, at which point it
correctly transitions the shard to SEALED with the right `sealed_lsn`.
Other changes in this commit:
- Remove dead commented-out LOGDEBUG in heap_chunk_selector.cpp.
- Bump version to 4.2.5 in conanfile.py.
## Tests
- **PGBlobIteratorSealedLsnCutoff (new)**: directly exercises both
branches of the fix:
- Case A — `snp_lsn` between `create_lsn` and `sealed_lsn`: asserts
the shard appears as OPEN with `sealed_lsn == INT64_MAX`.
- Case B — `snp_lsn >= sealed_lsn`: asserts the shard remains SEALED
with its original `sealed_lsn`.
- **PGBlobIterator (updated)**: the existing test seals a shard after
the snapshot LSN, so its state/sealed_lsn assertions are updated to
expect the downgraded OPEN state produced by the fix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## stable/v4.x #451 +/- ##
==============================================
Coverage ? 54.19%
==============================================
Files ? 36
Lines ? 5318
Branches ? 670
==============================================
Hits ? 2882
Misses ? 2143
Partials ? 293 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
yuwmao
reviewed
Jul 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
During snapshot generation, PGBlobIterator builds a shard list from the leader's in-memory PG state filtered by
upto_lsn(the snapshot LSN cutoff). Before this fix, a shard whosecreate_lsn <= upto_lsnwas always included with its current state — even if itssealed_lsnwas beyondupto_lsn(i.e., the seal happened after the snapshot point).When a follower received such a snapshot it saw the shard as SEALED and released the backing chunk. Log replay then tries to replay the
seal_shardentry for that LSN, but the chunk has already been reclaimed or relocated, causing the commit to fail with stale physical-chunk references.Root Cause
The race window is:
upto_lsn = L2.In step 4, S has
create_lsn = L1 <= L2so it passes the LSN filter, butsealed_lsn = L3 > L2. Old code included S with state SEALED. The follower then applied the snapshot, treated S as SEALED, released its chunk, and later failed when replaying the seal_shard log at L3.Fix
In
PGBlobIterator::PGBlobIterator, after thecreate_lsnfilter, add a second check:This converts the shard's snapshot state back to OPEN so the follower does not release the chunk prematurely. The follower will replay the
seal_shardlog entry when log replay catches up, at which point it correctly transitions the shard to SEALED with the rightsealed_lsn.Other changes in this commit:
Tests
snp_lsnbetweencreate_lsnandsealed_lsn: asserts the shard appears as OPEN withsealed_lsn == INT64_MAX.snp_lsn >= sealed_lsn: asserts the shard remains SEALED with its originalsealed_lsn.