Skip to content

fix(cache): drop cached packs that are no longer in the index - #578

Open
BradKollmyer wants to merge 1 commit into
rustic-rs:mainfrom
BradKollmyer:fix/cache-drop-absent-packs
Open

BradKollmyer wants to merge 1 commit into
rustic-rs:mainfrom
BradKollmyer:fix/cache-drop-absent-packs

Conversation

@BradKollmyer

@BradKollmyer BradKollmyer commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • When the index is loaded (to_indexed and to_indexed_ids, including backup), delete cached pack files whose ids are no longer in that index.
  • Packs that are still in the index stay. File size is not compared, matching restic's cache.Clear in prepareCache.
  • A failure to remove a cache file is logged and the command continues.

Why

A backup node never deletes packs itself. After another host prunes the repository, those pack files stayed in the local data/ cache forever. Restic drops them on the next index load. This does the same.

It does not shrink the cache down to the current parent snapshot. Packs the latest snapshot still references, and packs prune has not deleted yet, remain.

Note: backup-only hosts

This is the growth I am seeing on servers that only run backup.

A backup reads the parent snapshot's trees and stores each tree pack whole. It also stores every new tree pack it writes. Backup never removes anything from data/. After the pack cache on one of these hosts was emptied, the next successful backup wrote 507 pack files, 4.2 GB, all of them still present in the repository index. The index cache grew from 825 MB to 991 MB. The cache directory went from 851 MB to 5.2 GB.

This change does not remove those 507 packs. They are still in the index, so the sweep keeps them. It stops the cache from also keeping packs that a prune on another host has already deleted. Until that prune happens, a backup node's data/ cache keeps every tree pack it has read or written.

Validation

  • cargo test -p rustic_core --lib -- remove_ids_not_in_drops_only_absent_packs all_index_types
  • cargo test -p rustic_core --test integration index_load_drops_cached_packs_missing_from_the_index

The integration test uses a temporary cache directory, backs up one file, plants an extra data/ file, and checks that both to_indexed_ids() and to_indexed() remove it while keeping the tree pack the backup wrote.

Notes

Independent of #572 and the prune stack in #565–#569. Those stay as they are.

Clear pack cache entries missing from the index whenever it is loaded,
including backup. Packs another host has pruned no longer stay on a
backup node.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant