Repository navigation
Allow serving a pruned chain below deployed modules - #991
Merged
Merged
Conversation
🔍 Vulnerabilities of
|
| digest | sha256:f1b69c22ac9803cdf5766f0d1bc6401764375e98f6297da10dbcf7233c89d608 |
| vulnerabilities |     |
| platform | linux/amd64 |
| size | 126 MB |
| packages | 380 |
📦 Base Image ubuntu:24.04
| also known as |
|
| digest | sha256:f610ab94648195aa356059f5b41d6085c9d4d903c072430cdd1af7bdb646106b |
| vulnerabilities |    |
With Tier1Config.AllowStoreTruncation, modules whose initial block is below the first streamable block run from that block on, and their stores resume from snapshots taken before the chain was truncated.
sduchesneau
force-pushed
the
feature/truncating-store
branch
from
October 13, 2026 18:20
72a8acd to
f306f4f
Compare
With the linear handoff at the first streamable block, tier1 planned no store work and ran truncated stores from an empty state instead of their snapshot.
Stores without an initial block are named after the first streamable block, so raising it made tier1 resume from snapshots that tier2 could not load.
Refused requests return before tier1's detached last_used and spkg writes finish, so removing the test's temp dir failed now and then.
In dev mode, rounding the start block down to a segment boundary could land below it, where tier1 would load a snapshot and stream blocks that no longer exist.
A failed job stops the back filler, and jobs below the first streamable block, or starting from a snapshot named after another initial block, always fail.
Such a store is rebuilt from each new first streamable block while cached outputs still reflect its older state, so tier1 now asks for an explicit initial block.
sduchesneau
marked this pull request as ready for review
October 13, 2026 20:52
maoueh
reviewed
Oct 14, 2026
maoueh
left a comment
maoueh
left a comment
Contributor
There was a problem hiding this comment.
Something like AllowTailTruncation that would apply across the board, better error message for example.
The setting covers mappers as well as stores. Error messages now name the truncated tail and say how to fix the request.
sduchesneau
changed the title
A cursor replaces the start block, so a request with a cursor above the first streamable block was refused for a start block it did not use.
The start block was checked before the cursor was resolved on develop too, so the fix is part of this PR's changes.
sduchesneau
changed the title
maoueh
approved these changes
Oct 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Lets an operator prune the chain (raise
first-streamable-blockabove the initial block of deployed modules, mappers and stores alike) and keep serving every block that is still streamable.With
Tier1Config.AllowPrunedChain(forwarded to tier2 inProcessRangeRequest.allow_pruned_chain):0400100000-0350000000.kv), so tier1 resumes from the snapshots written before the chain was pruned and tier2 keeps writing0400101000-0350000000.kv, etc.InvalidArgument. The error names the store and says to set its initial block at or above the first streamable block, or to start the request where a later snapshot exists.Also fixed, with or without the setting, when the first streamable block is raised:
snapshot "…" does not exist. Those snapshots are now ignored and the store is rebuilt.Limitations:
The firehose-core flag (
--substreams-tier1-allow-pruned-chain) is in streamingfast/firehose-core#296.