perf: keep cached pack files open during prune tree walk - #566
Open
BradKollmyer wants to merge 18 commits into
Open
BradKollmyer wants to merge 18 commits into
BradKollmyer wants to merge 18 commits into
Conversation
OpenDAL only sends maxFileCount when ListOptions.limit is set. Without it, B2 defaults to 100 names per page, so prune/check pack listing becomes thousands of round-trips. Request 10000 for scheme b2.
256 KiB MAX_HOLESIZE split prune/restore pack reads into extra HTTP range GETs on high-latency stores. 4 MiB is cheaper than another RTT on a ~100 Mbps link.
Unused gaps in a pack were fetched one after another. Issue those range GETs in parallel; the packer already serializes writes.
stream_list used CPU-count Rayon workers and a zero-capacity channel, so reading index/snapshots was one GET at a time on high-latency backends. Use 16-32 workers with prefetch so GETs overlap decrypt/parse.
TreeStreamerOnce was hardcoded to 4 workers, which left B2 tree-pack GETs idle. Use 2x CPUs (8-32) with a larger out channel, and a HashSet for visited tree ids.
The used-id set was a BTreeMap, so inserts got slower as prune walked more blobs. A HashMap keeps that path O(1).
This was referenced Sep 7, 2026
A single Vec of blob ids doubles on growth. On a large repo that request is hundreds of MiB while the old buffer is still live, and musl aborts (openzwave: memory allocation of 807665664 bytes failed at reading index 906/1551). Store ids and full entries in 1 Mi-entry chunks and binary-search each sorted chunk.
Cache read_partial opened and closed the pack file for every tree blob. Keep up to 2048 FDs and pread from them. Hits skip LockPool and exists(); write/remove drop the cached handle.
Prune only needs file content ids and directory subtree ids. Deserialize a compact UsedBlobsTree instead of a full Vec<Node>, matching restic's streaming tree walk. Check and copy still load complete trees.
TreeStreamer dumped every snapshot root into a FIFO, so loaders jumped across packs. Keep a LIFO backlog and only feed one job per loader so consecutive trees tend to hit the same cached packs, matching restic.
getting packs was a serial B2 prefix list after finding used blobs. Start that list in the background so it finishes during the tree walk.
Overlapping the B2 pack list with index GETs made reading index... take minutes on a cold-ish server cache. List packs only during the tree walk so getting packs still hides without stalling the index.
16–32 parallel index readers help B2 RTTs but on a warm local cache they thrash disk. Use 2–4 workers when at least 75% of listed index/snapshot files already exist in the cache.
32 loaders on a 16-vCPU QEMU host spent most of finding used blobs... in KVM PV spinlocks and musl malloc. Use 4–8 loaders when the cache already has pack files; keep 2×CPUs (8–32) for a cold remote cache.
`i as u8` trips clippy::cast_possible_truncation and cast_sign_loss, which CI runs with -D warnings. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TdgDzcDaQVR4uqEPMDKZsu
rlimit/8 mapped Darwin ulimit 8192 to 1024 open pack files and added about 5s to prune getting-packs. Keep the 2048 cap and a 64-FD reserve.
nix getrlimit returns rlim_t, which is u32 on armv7.
BradKollmyer
force-pushed
the
perf/prune-walk
branch
from
September 7, 2026 20:41
de6d9f4 to
a9716cd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #564.
Stacked on #565. GitHub will also show that PR’s commits until it merges; review the unique range:
BradKollmyer/rustic_core@perf/prune-io...perf/prune-walk
pread, portable seek/read lock).stream_listworkers and tree loaders when pack files are cached.min(rlimit − 64, 2048)so a typical Darwin 8192 / linux 1024 soft limit does not collapse to 1024 or 128 FDs.