Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions CONTEXT.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,8 +20,8 @@ time, identified by a `timer_id`. The Timer is the *record*, not the message it
has a lifecycle (Scheduled → Due → Claimed → Committed) that its payload does not.

**Topic**:
The named channel a Timer is Scheduled on and that a subscriber consumes from. Two Redis keys are
derived per Topic; a Timer only ever exists on one.
The named channel a Timer is Scheduled on and that a subscriber consumes from. A Timer only ever
exists on one Topic.
_Avoid_: queue. A Topic is not a work queue: Timers on a Topic are
ordered by Activation time, not arrival, and a Claimed Timer stays on the Topic rather than leaving
it. `channel` survives only where FastStream owns the spelling: the `raw_message` field, the
Expand Down Expand Up @@ -53,8 +53,8 @@ excludes those.

**Claim**:
To take a Due Timer for processing by advancing its Activation time by the Lease, in one atomic
step, so no other worker sees it as Due. A Claim is not a lock: there is no lock key, no owner
recorded, and nothing to release. A contested Claim returns nothing and is simply skipped.
step, with unique ownership that allows only the current Claim to Commit. Scheduling a replacement
or Claiming again invalidates the previous ownership; a contested Claim returns nothing.

**Lease**:
The time window a Claim buys, `lease_ttl` seconds wide. It expires on its own; a worker that dies
Expand All @@ -64,5 +64,5 @@ exactly-once.

**Commit**:
To remove a Timer permanently once a handler has accounted for it, via `ack` or `reject`. `nack` is
not a Commit: it does nothing, and the Lease expiring is what causes redelivery. Cancellation
performs the identical removal, which is why the store exposes one `remove` and not two.
not a Commit: it does nothing, and the Lease expiring is what causes redelivery. A stale Commit
is a no-op; explicit cancellation removes the current Timer regardless of Claim ownership.
35 changes: 28 additions & 7 deletions docs/introduction/how-it-works.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,20 @@

## Redis data structures

`faststream-redis-timers` uses two Redis keys **per topic** to store scheduled timers:
`faststream-redis-timers` uses three Redis keys **per topic** to store scheduled timers:

| Key pattern | Type | Purpose |
|-------------|------|---------|
| `timers_timeline:{topic}` | Sorted set | Maps each `timer_id` to its activation Unix timestamp (score) |
| `timers_payloads:{topic}` | Hash | Maps each `timer_id` to its serialized message body |
| `timers_claims:{digest}` by default | Hash | Maps each claimed `timer_id` to its current ownership token |

When you publish a timer for topic `invoices` with `timer_id` `abc-123`, both keys are written atomically using a Redis pipeline:
When you publish a timer for topic `invoices` with `timer_id` `abc-123`, the timer is written and any previous claim is invalidated atomically using a Redis pipeline:

```
ZADD timers_timeline:invoices <activation_ts> abc-123
HSET timers_payloads:invoices abc-123 <encoded_envelope>
HDEL <ownership_key> abc-123
```

The envelope is FastStream's `BinaryMessageFormatV1`, a binary-safe wire format that carries the message body alongside `correlation_id`, `content-type`, `reply_to`, and any user-supplied headers. The same format is used by FastStream's built-in Redis broker.
Expand All @@ -26,26 +28,28 @@ Each subscriber runs a background polling loop that:
2. For each due timer, spawns a task (bounded by the `max_concurrent` capacity limiter) that runs an atomic Lua **claim** script that:
- Verifies the timer is still due (score ≤ now)
- Pushes its score forward by `lease_ttl` seconds, which grants the worker a lease
- Returns the payload
- Records a fresh unique ownership token and returns the payload
3. Delivers the payload to the user handler
4. On handler success, runs an atomic Lua **commit** script (`ZREM` + `HDEL`) to remove the timer from Redis
4. On handler success, runs an atomic Lua **commit** script that removes the timer and its claim token only if the captured token still matches
5. On handler exception, leaves the timer in place; the lease eventually expires and another worker re-claims it

This is the standard SQS-style **visibility-timeout** pattern. The timer's own score in the sorted set is the lease deadline; there is no separate lock primitive.

## At-least-once delivery

The lease ensures each due timer is processed by exactly one worker at a time. Because the timer is removed from Redis only after the handler completes successfully, **no timer is lost on crash**: if the worker dies mid-handler, the lease expires and the timer is re-delivered.
An atomic claim gives one worker ownership until another successful claim or a replacement invalidates it. Lease expiry permits another worker to claim while the original handler may still be running. Because the timer is removed from Redis only after the handler completes successfully, **no timer is lost on crash**: if the worker dies mid-handler, the lease expires and the timer is re-delivered.

The trade-off: handlers that take longer than `lease_ttl`, or workers that crash after the handler ran but before the commit landed, may see the timer delivered more than once. Handlers must therefore be **idempotent**.

## Ack / Nack / Reject

| Action | Effect |
|--------|--------|
| `ack` | Atomically removes the timer from `timers_timeline` and `timers_payloads` |
| `ack` | Removes the timer and claim metadata only if this delivery still owns the claim |
| `nack` | No-op; the lease expires and another worker re-claims the timer |
| `reject` | Same as `ack`: permanently removes the timer (use for poison-pill messages) |
| `reject` | Same conditional removal as `ack` (use for poison-pill messages) |

Replacement, cancel/recreate, and lease takeover make a previous delivery's completion a successful no-op. Corrupt-payload cleanup uses the same ownership check. Explicit cancellation remains unconditional. Successful commits, cancellation, and bulk cancellation remove claim metadata.

The default ack policy is `NACK_ON_ERROR`: the timer is acknowledged on success, and left for retry on any unhandled exception.

Expand All @@ -57,6 +61,7 @@ The default ack policy is `NACK_ON_ERROR`: the timer is acknowledged on success,
|-----------|---------|-------------|
| `timeline_key` | `timers_timeline` | Prefix for sorted set keys (`{timeline_key}:{topic}`) |
| `payloads_key` | `timers_payloads` | Prefix for hash keys (`{payloads_key}:{topic}`) |
| `claims_key` | `None` | Optional ownership hash prefix (`{claims_key}:{topic}`); by default the key is derived from the timeline and payload keys |
| `start_timeout` | `3.0` s | Max wait for the first Redis ping during startup |

### Per-subscriber (`@broker.subscriber("topic", ...)`)
Expand All @@ -69,6 +74,22 @@ The default ack policy is `NACK_ON_ERROR`: the timer is acknowledged on success,
| `lease_ttl` | `30` s | How long a worker holds the lease before another worker may re-claim. Set to ~3–5× the P99 handler runtime: lower values speed up recovery from worker death, higher values tolerate handler GC pauses and clock skew |
| `command_timeout` | `None` (`lease_ttl`) | Most time the subscriber waits on one Redis command (fetch, claim, ack). A primary that goes silent without closing its connections otherwise stalls the subscriber; redis-py before 8.0 sets no socket timeout by default. Must be > 0 and <= `lease_ttl` |

Redis client encoding, encoding error handling, and `decode_responses` settings are preserved. Timer IDs use the client's encoder for storage and inspection; binary message envelopes remain raw bytes during claims. During polling, IDs are fetched as raw bytes regardless of response decoding. IDs that cannot decode and re-encode to the same bytes are conditionally cleaned using their original byte identity, so lossy decoding cannot redirect claims or block following valid timers.

By default, ownership keys are `timers_claims:{digest}`. The digest is SHA-256 of the full timeline and payload key bytes as encoded by the client, each preceded by its eight-byte big-endian length. This gives independently configured stores separate ownership metadata, remains stable across restarts, and requires no reserved binary namespace. An explicit `claims_key` instead uses `{claims_key}:{topic}`, including any topic prefix.

Timeline, payload, and ownership keys for a topic must be distinct after Redis encoding. Mutating operations reject overlapping keys with `ValueError` before issuing Redis commands. Use `claims_key` to select another namespace when needed. Across independently configured brokers, keep all key namespaces non-overlapping; workers sharing timers must use the same ownership configuration. Changing ownership configuration while workers are active is unsafe.

## Upgrading existing timers

Existing timer payloads and scores need no rewrite. Their first successful claim records a fresh ownership token; existing leases remain unavailable until their stored deadline.

Stop every old process that writes to the namespace before starting fenced workers. This includes consumers and their in-flight handlers, publishing-only processes, and cancellation clients. Upgrade all these writers before resuming them. Old consumers delete unconditionally; old publishers replace payloads and deadlines without invalidating claim ownership, allowing a fenced worker's stale completion to delete the replacement. Old cancellation clients also leave claim metadata behind. Stopping only old consumers is insufficient.

Persisted records may stay in Redis during this coordinated transition; new workers adopt them on claim. If old writers must remain active, use a separate namespace for fenced writers and workers. Downgrading any writer to the unfenced protocol restores the compatibility risk.

Fencing protects Redis timer state. It cannot retract external effects from a handler whose lease expired. Keep handlers idempotent; this provides at-least-once delivery and does not guarantee exactly-once publication to Kafka or other systems. Leases are not renewed automatically.

## Operational requirements

`faststream-redis-timers` is designed for a **single-primary Redis** (Sentinel-managed primary/replica setups are supported; Redis Cluster is not: constructing `TimersBroker` with a `RedisCluster` client raises `TypeError`).
Expand Down
1 change: 1 addition & 0 deletions docs/usage/basic.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,7 @@ async def schedule_reminder() -> None:
| `client` | _required for production_ | `redis.asyncio.Redis` client instance; the caller owns its lifecycle (the broker does not close it). The constructor accepts `None`, but a broker without a client raises `IncorrectState` on the first operation; `None` is the test-broker shape (see [Testing](./testing.md)). |
| `timeline_key` | `timers_timeline` | Sorted set key prefix; each topic uses `{timeline_key}:{topic}` |
| `payloads_key` | `timers_payloads` | Hash key prefix; each topic uses `{payloads_key}:{topic}` |
| `claims_key` | `None` | Optional ownership hash prefix; by default derived from the encoded timeline and payload key identities. An explicit prefix uses `{claims_key}:{topic}`. Keep namespaces distinct. |
| `start_timeout` | `3.0` | Seconds to wait for the subscriber's first Redis ping during startup |
| `graceful_timeout` | `15.0` | Seconds to wait for in-flight timers on shutdown |

Expand Down
3 changes: 2 additions & 1 deletion faststream_redis_timers/broker.py
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,7 @@ def __init__( # noqa: PLR0913
*,
timeline_key: str = "timers_timeline",
payloads_key: str = "timers_payloads",
claims_key: str | None = None,
start_timeout: float = 3.0,
decoder: CustomCallable | None = None,
parser: CustomCallable | None = None,
Expand All @@ -142,7 +143,7 @@ def __init__( # noqa: PLR0913
raise TypeError(msg)
fd_config = FastDependsConfig(use_fastdepends=apply_types)
connection = ConnectionState(client)
store = TimerStore(connection, timeline_key, payloads_key)
store = TimerStore(connection, timeline_key, payloads_key, claims_key)
broker_config = TimersBrokerConfig(
connection=connection,
start_timeout=start_timeout,
Expand Down
5 changes: 3 additions & 2 deletions faststream_redis_timers/message.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,17 +12,18 @@ class TimerMessage(typing.TypedDict):
channel: str
timer_id: str
data: bytes
claim_token: typing.NotRequired[str]


class TimerStreamMessage(StreamMessage["TimerMessage"]):
"""Stream message that removes the timer from Redis only on ack/reject.

Lease-based at-least-once delivery: the timer remains in the timeline
(with its score pushed forward by `lease_ttl`) while the handler runs.
`ack()` and `reject()` atomically remove it; `nack()` is a no-op so the
`ack()` and `reject()` remove it only while this delivery owns the claim; `nack()` is a no-op so the
lease expires and another worker re-claims the timer.

A timer whose payload cannot be parsed is removed and raises on `decode()`,
A delivery whose payload cannot be parsed conditionally commits and raises on `decode()`,
so the handler never runs.
"""

Expand Down
19 changes: 11 additions & 8 deletions faststream_redis_timers/parser/parser.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,12 +20,13 @@ def __init__(self, config: "TimersSubscriberConfig") -> None:

async def parse_message(self, msg: "TimerMessage") -> TimerStreamMessage:
timer_id = msg["timer_id"]
token = msg.get("claim_token")
discard_corrupt = None
try:
body, headers = TimerMessageFormat.parse(msg["data"])
except ValueError as e:
body, headers = msg["data"], {}
discard_corrupt = partial(self._discard_corrupt, msg, e)
discard_corrupt = partial(self._discard_corrupt, timer_id, token, len(msg["data"]), e)
return TimerStreamMessage(
raw_message=msg,
body=body,
Expand All @@ -34,21 +35,23 @@ async def parse_message(self, msg: "TimerMessage") -> TimerStreamMessage:
message_id=timer_id,
correlation_id=headers.get("correlation_id", timer_id),
reply_to=headers.get("reply_to", ""),
_remove=partial(self._remove, timer_id),
_remove=partial(self._remove, timer_id, token),
_discard_corrupt=discard_corrupt,
)

async def _remove(self, timer_id: str) -> None:
async def _remove(self, timer_id: str, token: str | None) -> None:
if token is None:
return
store = self._config._outer_config.store # noqa: SLF001
with anyio.fail_after(self._config.timer_sub.effective_command_timeout):
await store.remove(self._config.full_topic, timer_id)
await store.commit(self._config.full_topic, timer_id, token)

async def _discard_corrupt(self, msg: "TimerMessage", error: ValueError) -> typing.NoReturn:
async def _discard_corrupt(self, timer_id: str, token: str | None, size: int, error: ValueError) -> typing.NoReturn:
outer_config = self._config._outer_config # noqa: SLF001
await self._remove(msg["timer_id"])
await self._remove(timer_id, token)
outer_config.logger.log(
f"Timer {msg['timer_id']!r} on {self._config.full_topic!r} removed: "
f"its {len(msg['data'])}-byte payload cannot be parsed",
f"Timer {timer_id!r} on {self._config.full_topic!r} removed if still owned: "
f"its {size}-byte payload cannot be parsed",
logging.ERROR,
exc_info=error,
)
Expand Down
Loading
Loading