fix(cache): drop cached packs that are no longer in the index - #578
Open
BradKollmyer wants to merge 1 commit into
Open
BradKollmyer wants to merge 1 commit into
BradKollmyer wants to merge 1 commit into
Conversation
Clear pack cache entries missing from the index whenever it is loaded, including backup. Packs another host has pruned no longer stay on a backup node.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
to_indexedandto_indexed_ids, including backup), delete cached pack files whose ids are no longer in that index.cache.ClearinprepareCache.Why
A backup node never deletes packs itself. After another host prunes the repository, those pack files stayed in the local
data/cache forever. Restic drops them on the next index load. This does the same.It does not shrink the cache down to the current parent snapshot. Packs the latest snapshot still references, and packs prune has not deleted yet, remain.
Note: backup-only hosts
This is the growth I am seeing on servers that only run backup.
A backup reads the parent snapshot's trees and stores each tree pack whole. It also stores every new tree pack it writes. Backup never removes anything from
data/. After the pack cache on one of these hosts was emptied, the next successful backup wrote 507 pack files, 4.2 GB, all of them still present in the repository index. The index cache grew from 825 MB to 991 MB. The cache directory went from 851 MB to 5.2 GB.This change does not remove those 507 packs. They are still in the index, so the sweep keeps them. It stops the cache from also keeping packs that a prune on another host has already deleted. Until that prune happens, a backup node's
data/cache keeps every tree pack it has read or written.Validation
cargo test -p rustic_core --lib -- remove_ids_not_in_drops_only_absent_packs all_index_typescargo test -p rustic_core --test integration index_load_drops_cached_packs_missing_from_the_indexThe integration test uses a temporary cache directory, backs up one file, plants an extra
data/file, and checks that bothto_indexed_ids()andto_indexed()remove it while keeping the tree pack the backup wrote.Notes
Independent of #572 and the prune stack in #565–#569. Those stay as they are.