Repack chunks index into entry-count-bounded fragments

[ThomasWaldmann]

What

Replace the unconditional "collapse all index/* fragments into one all-in-one index" behavior with an entry-count-based repacking policy that keeps each chunks-index fragment within [CHUNKINDEX_FRAGMENT_ENTRIES_MIN, CHUNKINDEX_FRAGMENT_ENTRIES_MAX] entries where possible.

Why

The chunks index is stored in the repo as immutable, content-addressed index/<sha256> fragments. Each backup appends a small incremental fragment. Several paths previously consolidated all fragments into a single object and deleted the rest, which is problematic:

• a single index for a large repo can be huge; writing/reading it is all-or-nothing;
• deleting every fragment and replacing it with one object is hostile to a future borgstore writethrough cache on index/: it would invalidate the cached fragments of every client of the repo, forcing a full re-download.

Bounding fragment size (upper bound) and count (lower bound), and keeping large fragments immutable/stable, is groundwork for that future caching while also making the current behavior cheaper.

How

• write_chunkindex_to_repo() now always splits its output into fragments of at most MAX entries (streamed, so memory stays bounded), collecting all new hashes so delete_other/delete_these never delete a fragment it just wrote.
• repack_chunkindex() merges small (< MIN) fragments, using deferred merging: it only acts when the small fragments can seal a full fragment (sum >= MIN) or too many have piled up (> CHUNKINDEX_SMALL_FRAGMENT_CAP). Fragments already in range are left untouched (immutable).
• Wired into AdHocWithFilesCache.close() and the build_chunkindex_from_repo merge>1 branch (which no longer collapses-and-deletes everything).
• list_chunkindex_fragments() estimates a fragment's entry count from its stored byte size (~CHUNKINDEX_ENTRY_SIZE bytes/entry), so fragments are classified without loading them.
• Deletion of superseded fragments is gated on the replacement fragments being present in the repo (their content hashes), not on a fresh upload having happened. Combined with repack_chunkindex() using force_write=False, this makes repack idempotent: a merged fragment that already exists is dedupe-skipped rather than re-uploaded, and the small sources are still deleted even if a previous repack crashed after storing but before deleting. close() also treats repack as best-effort — a transient failure warns instead of failing an already-committed backup.

Note: write_chunkindex_to_repo now splits the incremental path too (not just full rewrites), so even a huge initial backup's single incremental fragment stays bounded by MAX.

Not in scope

Enabling the index/ writethrough cache — deferred.

Tests

New unit tests in cache_test.py cover: full-write and incremental-write splitting, the deferred-merge triggers (defer / seal-at-MIN / cap-forced), sealed-fragment survival across a repack, identical reconstruction via build_chunkindex_from_repo, an end-to-end multi-session test confirming close() consolidates fragments across runs, and an idempotent-repack test asserting a repack whose merged fragment already exists (dedupe-skipped) still deletes the small sources. Existing repository / compact / check / create / delete / prune suites pass.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 91.91919% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.14%. Comparing base (5aaa2dd) to head (c4a62b4).
⚠️ Report is 9 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/borg/cache.py	91.57%	5 Missing and 3 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #9846      +/-   ##
==========================================
+ Coverage   85.11%   85.14%   +0.03%     
==========================================
  Files          93       93              
  Lines       15408    15483      +75     
  Branches     2326     2338      +12     
==========================================
+ Hits        13114    13183      +69     
- Misses       1596     1600       +4     
- Partials      698      700       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

[ThomasWaldmann]

@mr-raj12 please review ^^^

[mr-raj12]

@mr-raj12 please review ^^^

Network outage, give me EOD's time