Deferred from the #285 (LXUD v2 PR2) review as pre-existing + low severity.
Problem
The no-op-deletion skip in LexUserHistory::apply_records gates the Tombstone append (and the scrub) on UserHistory::contains_entries, which reads only the in-memory unigram/bigram maps. An entry can be absent from memory (evicted by evict_map once max_unigrams is exceeded) yet still present in the last on-disk checkpoint — a checkpoint is a snapshot from an earlier compaction, and entries evicted after that snapshot linger in the checkpoint until the next compaction re-snapshots and truncates.
For such an entry, contains_entries returns false → no Tombstone frame is written and no scrub compaction is posted. The user's ForwardDelete is silently a no-op. If the process crashes / restarts before a later compaction rewrites the checkpoint, open_recovering loads the stale checkpoint (which still contains the entry) and, with no tombstone to neutralize it, the "deleted" conversion resurrects.
Reachability (why low)
Requires the conjunction of: near-cap history (heavy long-term use to reach max_unigrams = 10000), the deleted entry being in the bounded, actively-shrinking evicted-but-not-yet-recompacted window, and a crash inside that window.
Pre-existing
Not introduced by PR2. contains_entries (added in PR2) preserves the same memory-only semantics as v1's remove_entries, which likewise returned false for evicted entries and never persisted the deletion. The contains_entries ⇔ remove_entries equivalence is intact (both consult only memory, pinned by a test); the gap is that both ignore the on-disk checkpoint.
Possible fix
Emit a Tombstone whenever the pair could be present in the durable checkpoint (not only when it is in memory), or gate the no-op skip on "not in memory and not in the last checkpoint". Either restores the invariant that a requested deletion survives a restart.
Severity: low. Files: engine/crates/lex-core/src/user_history/mod.rs (contains_entries / remove_entries / evict), engine/src/api/resources.rs (apply_records).
Deferred from the #285 (LXUD v2 PR2) review as pre-existing + low severity.
Problem
The no-op-deletion skip in
LexUserHistory::apply_recordsgates the Tombstone append (and the scrub) onUserHistory::contains_entries, which reads only the in-memory unigram/bigram maps. An entry can be absent from memory (evicted byevict_maponcemax_unigramsis exceeded) yet still present in the last on-disk checkpoint — a checkpoint is a snapshot from an earlier compaction, and entries evicted after that snapshot linger in the checkpoint until the next compaction re-snapshots and truncates.For such an entry,
contains_entriesreturnsfalse→ no Tombstone frame is written and no scrub compaction is posted. The user's ForwardDelete is silently a no-op. If the process crashes / restarts before a later compaction rewrites the checkpoint,open_recoveringloads the stale checkpoint (which still contains the entry) and, with no tombstone to neutralize it, the "deleted" conversion resurrects.Reachability (why low)
Requires the conjunction of: near-cap history (heavy long-term use to reach
max_unigrams= 10000), the deleted entry being in the bounded, actively-shrinking evicted-but-not-yet-recompacted window, and a crash inside that window.Pre-existing
Not introduced by PR2.
contains_entries(added in PR2) preserves the same memory-only semantics as v1'sremove_entries, which likewise returnedfalsefor evicted entries and never persisted the deletion. Thecontains_entries⇔remove_entriesequivalence is intact (both consult only memory, pinned by a test); the gap is that both ignore the on-disk checkpoint.Possible fix
Emit a Tombstone whenever the pair could be present in the durable checkpoint (not only when it is in memory), or gate the no-op skip on "not in memory and not in the last checkpoint". Either restores the invariant that a requested deletion survives a restart.
Severity: low. Files:
engine/crates/lex-core/src/user_history/mod.rs(contains_entries/remove_entries/evict),engine/src/api/resources.rs(apply_records).