perf: stream-decode prune trees without serde node parse by BradKollmyer · Pull Request #567 · rustic-rs/rustic_core

BradKollmyer · 2026-09-07T00:32:53Z

Part of #564.

Stacked on #566. Review the unique range:

BradKollmyer/rustic_core@perf/prune-walk...perf/prune-json

Reuse a zstd decompressor per thread for blob reads.
Decode prune tree blob ids with a hex nibble table.
Record used blob ids from loader threads; fill per-loader maps.
Stream-decode prune tree nodes without full node structs.
Use FxHash for used-blob maps and the tree visited set.
Scan prune tree JSON for type/content/subtree without serde of unused node fields.
Preserve escaped JSON references; unknown node type is an error.

OpenDAL only sends maxFileCount when ListOptions.limit is set. Without it, B2 defaults to 100 names per page, so prune/check pack listing becomes thousands of round-trips. Request 10000 for scheme b2.

256 KiB MAX_HOLESIZE split prune/restore pack reads into extra HTTP range GETs on high-latency stores. 4 MiB is cheaper than another RTT on a ~100 Mbps link.

Unused gaps in a pack were fetched one after another. Issue those range GETs in parallel; the packer already serializes writes.

stream_list used CPU-count Rayon workers and a zero-capacity channel, so reading index/snapshots was one GET at a time on high-latency backends. Use 16-32 workers with prefetch so GETs overlap decrypt/parse.

TreeStreamerOnce was hardcoded to 4 workers, which left B2 tree-pack GETs idle. Use 2x CPUs (8-32) with a larger out channel, and a HashSet for visited tree ids.

The used-id set was a BTreeMap, so inserts got slower as prune walked more blobs. A HashMap keeps that path O(1).

A single Vec of blob ids doubles on growth. On a large repo that request is hundreds of MiB while the old buffer is still live, and musl aborts (openzwave: memory allocation of 807665664 bytes failed at reading index 906/1551). Store ids and full entries in 1 Mi-entry chunks and binary-search each sorted chunk.

Cache read_partial opened and closed the pack file for every tree blob. Keep up to 2048 FDs and pread from them. Hits skip LockPool and exists(); write/remove drop the cached handle.

Prune only needs file content ids and directory subtree ids. Deserialize a compact UsedBlobsTree instead of a full Vec<Node>, matching restic's streaming tree walk. Check and copy still load complete trees.

TreeStreamer dumped every snapshot root into a FIFO, so loaders jumped across packs. Keep a LIFO backlog and only feed one job per loader so consecutive trees tend to hit the same cached packs, matching restic.

getting packs was a serial B2 prefix list after finding used blobs. Start that list in the background so it finishes during the tree walk.

Overlapping the B2 pack list with index GETs made reading index... take minutes on a cold-ish server cache. List packs only during the tree walk so getting packs still hides without stalling the index.

16–32 parallel index readers help B2 RTTs but on a warm local cache they thrash disk. Use 2–4 workers when at least 75% of listed index/snapshot files already exist in the cache.

32 loaders on a 16-vCPU QEMU host spent most of finding used blobs... in KVM PV spinlocks and musl malloc. Use 4–8 loaders when the cache already has pack files; keep 2×CPUs (8–32) for a cold remote cache.

`i as u8` trips clippy::cast_possible_truncation and cast_sign_loss, which CI runs with -D warnings. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TdgDzcDaQVR4uqEPMDKZsu

rlimit/8 mapped Darwin ulimit 8192 to 1024 open pack files and added about 5s to prune getting-packs. Keep the 2048 cap and a 64-FD reserve.

nix getrlimit returns rlim_t, which is u32 on armv7.

decode_all created a DCtx for every compressed tree blob. During prune finding used blobs that showed up as ZSTD_createDCtx, munmap, and ~60k page faults/s. Keep one bulk Decompressor in thread-local storage.

hex::FromHex<&str> was ~6% of finding used blobs after zstd reuse. Parse the 64-char ids with a lookup table during UsedBlobsTree JSON decode instead.

A single consumer HashMap::insert serialized the used-blob walk. Loaders now insert into a 16-way sharded map while they decode trees.

A derived UsedBlobNode still paid serde skip/parse of every field. Visit only type, content, and subtree and push ids straight into UsedBlobsTree.

Sharded Mutex HashMaps still spent ~35% on insert and lock contention. Each tree loader now owns a HashMap and the maps are merged after the walk.

siphash HashMap insert of BlobId was ~27% of finding used blobs after locks were removed. FxHash is enough for in-memory blob ids.

Unused node fields (names, mtime, xattrs) were still UTF-8-validated and skipped through serde. A dedicated restic-tree scanner only hex-decodes file content and dir subtree ids.

BradKollmyer added 6 commits September 6, 2026 14:49

fix: request B2's maximum list page size

9210d41

OpenDAL only sends maxFileCount when ListOptions.limit is set. Without it, B2 defaults to 100 names per page, so prune/check pack listing becomes thousands of round-trips. Request 10000 for scheme b2.

fix: fetch 4 MiB unused pack gaps in one read

d468d44

256 KiB MAX_HOLESIZE split prune/restore pack reads into extra HTTP range GETs on high-latency stores. 4 MiB is cheaper than another RTT on a ~100 Mbps link.

fix: range-GET prune repack chunks in parallel

a75f3c8

Unused gaps in a pack were fetched one after another. Issue those range GETs in parallel; the packer already serializes writes.

fix: prefetch index files with extra IO workers

b99e719

stream_list used CPU-count Rayon workers and a zero-capacity channel, so reading index/snapshots was one GET at a time on high-latency backends. Use 16-32 workers with prefetch so GETs overlap decrypt/parse.

fix: scale tree loaders during prune used-blob walk

ec07d6a

TreeStreamerOnce was hardcoded to 4 workers, which left B2 tree-pack GETs idle. Use 2x CPUs (8-32) with a larger out channel, and a HashSet for visited tree ids.

fix: store prune used blob ids in a HashMap

43365be

The used-id set was a BTreeMap, so inserts got slower as prune walked more blobs. A HashMap keeps that path O(1).

This was referenced Sep 7, 2026

perf: collect prune used ids in sharded maps after the walk #568

Open

perf: speed up prune used-blob walk on a warm pack cache #564

Open

BradKollmyer force-pushed the perf/prune-json branch from 6583760 to b005297 Compare September 7, 2026 01:02

BradKollmyer and others added 20 commits September 7, 2026 13:40

fix: keep cached pack files open for tree blob reads

0464477

Cache read_partial opened and closed the pack file for every tree blob. Keep up to 2048 FDs and pread from them. Hits skip LockPool and exists(); write/remove drop the cached handle.

fix: stream-decode trees during prune used-blob walk

4674695

Prune only needs file content ids and directory subtree ids. Deserialize a compact UsedBlobsTree instead of a full Vec<Node>, matching restic's streaming tree walk. Check and copy still load complete trees.

fix: walk unique trees depth-first

721dc03

TreeStreamer dumped every snapshot root into a FIFO, so loaders jumped across packs. Keep a LIFO backlog and only feed one job per loader so consecutive trees tend to hit the same cached packs, matching restic.

fix: list pack files during prune index and tree walk

c65570c

getting packs was a serial B2 prefix list after finding used blobs. Start that list in the background so it finishes during the tree walk.

fix: start prune pack listing after the index is loaded

5e7996f

Overlapping the B2 pack list with index GETs made reading index... take minutes on a cold-ish server cache. List packs only during the tree walk so getting packs still hides without stalling the index.

fix: use fewer stream_list workers on a warm cache

b664070

16–32 parallel index readers help B2 RTTs but on a warm local cache they thrash disk. Use 2–4 workers when at least 75% of listed index/snapshot files already exist in the cache.

fix: use fewer tree loaders when pack files are cached

188e5bc

32 loaders on a 16-vCPU QEMU host spent most of finding used blobs... in KVM PV spinlocks and musl malloc. Use 4–8 loaders when the cache already has pack files; keep 2×CPUs (8–32) for a cold remote cache.

fix: build cache test payloads without lossy int casts

4ef023f

`i as u8` trips clippy::cast_possible_truncation and cast_sign_loss, which CI runs with -D warnings. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TdgDzcDaQVR4uqEPMDKZsu

fix: bound cached handles and synchronize portable range reads

34d586c

fix: size cached pack handles from rlimit minus a reserve

3631eab

rlimit/8 mapped Darwin ulimit 8192 to 1024 open pack files and added about 5s to prune getting-packs. Keep the 2048 cap and a 64-FD reserve.

fix: convert rlimit to u64 for 32-bit prune cache cap

a9716cd

nix getrlimit returns rlim_t, which is u32 on armv7.

fix: reuse a zstd decompressor per thread for blob reads

be68643

decode_all created a DCtx for every compressed tree blob. During prune finding used blobs that showed up as ZSTD_createDCtx, munmap, and ~60k page faults/s. Keep one bulk Decompressor in thread-local storage.

fix: decode prune tree blob ids with a hex nibble table

7961821

hex::FromHex<&str> was ~6% of finding used blobs after zstd reuse. Parse the 64-char ids with a lookup table during UsedBlobsTree JSON decode instead.

fix: record prune used blob ids from loader threads

186a925

A single consumer HashMap::insert serialized the used-blob walk. Loaders now insert into a 16-way sharded map while they decode trees.

fix: stream-decode prune tree nodes without full node structs

19c41bd

A derived UsedBlobNode still paid serde skip/parse of every field. Visit only type, content, and subtree and push ids straight into UsedBlobsTree.

fix: fill prune used blob ids in per-loader maps

9aa8740

Sharded Mutex HashMaps still spent ~35% on insert and lock contention. Each tree loader now owns a HashMap and the maps are merged after the walk.

fix: use FxHash for prune used-blob maps and tree visited set

461977d

siphash HashMap insert of BlobId was ~27% of finding used blobs after locks were removed. FxHash is enough for in-memory blob ids.

fix: scan prune tree JSON without serde node field parse

8d37cf5

Unused node fields (names, mtime, xattrs) were still UTF-8-validated and skipped through serde. A dedicated restic-tree scanner only hex-decodes file content and dir subtree ids.

fix: preserve escaped JSON references during prune

b29a5cd

BradKollmyer force-pushed the perf/prune-json branch from b005297 to b29a5cd Compare September 7, 2026 20:41

Navigation Menu

perf: stream-decode prune trees without serde node parse - #567

perf: stream-decode prune trees without serde node parse#567
BradKollmyer wants to merge 26 commits into
rustic-rs:mainfrom
BradKollmyer:perf/prune-json

BradKollmyer commented Sep 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

BradKollmyer commented Sep 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant