Skip to content

fix: fence timer completion against replacement and lease takeover - #126

Merged
lesnik512 merged 3 commits into
mainfrom
fix/fenced-timer-commits
Nov 14, 2026
Merged

lesnik512 merged 3 commits into
mainfrom
fix/fenced-timer-commits

Conversation

@lesnik512

@lesnik512 lesnik512 commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

A handler completing an earlier timer could delete a same-ID replacement or a timer reclaimed by another worker after lease expiry. Give every successful claim a unique ownership token and make ack, reject, and corrupt-payload cleanup conditional on that captured token. Scheduling invalidates earlier ownership; explicit cancellation remains unconditional.

Ownership metadata is removed on valid completion, cancellation, bulk cancellation, and orphan cleanup. Existing payloads and deadlines need no rewrite: persisted timers acquire metadata when claimed. Application migration changes are outside this PR.

Redis client settings and ownership storage

Redis client encoding, encoding error handling, and decode_responses settings remain under caller control. Timer ID inspection uses the client's encoder, and claims preserve binary payload bytes. Regression coverage includes Latin-1, UTF-8 with surrogateescape, ASCII with replacement errors, and both response-decoding modes. Polling fetches raw timer IDs independently of response decoding and checks decode/encode byte identity. Unrepresentable IDs are conditionally cleaned by their original bytes, preventing lossy decoding from redirecting claims or blocking later valid timers.

Default ownership keys use a SHA-256 digest of the encoded timeline and payload key identities, each prefixed with its eight-byte length. This separates independently configured stores without reserving a binary key namespace. The optional claims_key parameter selects an explicit {claims_key}:{topic} namespace.

Mutating operations verify that timeline, payload, and ownership key bytes are distinct before issuing Redis commands. Independently configured brokers must use non-overlapping namespaces; workers sharing timers must use the same ownership configuration.

Upgrade compatibility

Stop all old writers, including consumers and in-flight handlers, publishing-only processes, and cancellation clients, before fenced workers use the same namespace. Upgrade every writer before resuming it, or isolate old and new deployments in separate namespaces. Old producers do not invalidate ownership, and old consumers delete unconditionally. Do not change ownership configuration while workers are active.

Delivery remains at-least-once. Fencing protects timer state, not external side effects; handlers must remain idempotent.

Validation

  • Real-Redis tests reproduced stale ack/reject deletion and the earlier encoding restrictions before their fixes. Mutation checks verified detection of missing fencing, lease checks, orphan-metadata cleanup, and encoded-key collision validation.
  • New real-Redis ignore/replace regressions failed before the raw-ID fix in both response-decoding modes and now verify eventual valid delivery plus preservation of a nearby alias ID. Simultaneous-owner isolation tests reject a shared-default-key mutation; cross-encoding peers reject hashing unencoded key names.
  • Full suite on Python 3.11: 244 passed, 100% coverage (just test-ci against Redis 8). Four existing mock coroutine warnings remain.
  • All 64 fencing tests passed with the declared dependency floors, redis==5.0.1 and faststream==0.7.6, in an isolated environment.
  • just lint-ci: formatting, lint, and type checks passed.
  • just docs-build and just adr-check passed.
  • just test-dr: both Sentinel primary-kill and primary-pause failover drills passed.
  • git diff --check passed.

@lesnik512
lesnik512 merged commit be3b195 into main Nov 14, 2026
17 checks passed
@lesnik512
lesnik512 deleted the fix/fenced-timer-commits branch November 14, 2026 09:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant