|
| 1 | +# Transactional notification delivery with PostgreSQL |
| 2 | + |
| 3 | +`PgNotificationOutbox` is a separate opt-in table and worker for non-task |
| 4 | +notifications. Install `adcp[pg]` and prepare the notification using the |
| 5 | +[preparation API](prepared-notifications.md). Existing task and reporting outbox |
| 6 | +contracts and migrations are independent. |
| 7 | + |
| 8 | +```python |
| 9 | +from adcp import PgNotificationOutbox, WebhookSender |
| 10 | + |
| 11 | +# Use the same pool and connection as the business operation. Encryption key |
| 12 | +# material comes from your secret manager and must survive process restarts. |
| 13 | +outbox = PgNotificationOutbox( |
| 14 | + pool=pool, |
| 15 | + sender=sender, # SDK-owned RFC 9421 sender, or sender_resolver=resolver |
| 16 | + encryption_key=notification_encryption_key, # exactly 32 bytes |
| 17 | + delivery_retry_horizon_seconds=86400, |
| 18 | +) |
| 19 | +await outbox.create_schema() |
| 20 | + |
| 21 | +prepared = WebhookSender.prepare_account_status_changed( |
| 22 | + url="https://buyer.example/hooks", |
| 23 | + payload={ |
| 24 | + "notification_id": "account-transition-42", |
| 25 | + "subscriber_id": "buyer-primary", |
| 26 | + "account_id": "acct_1", |
| 27 | + "previous_status": "pending_approval", |
| 28 | + "status": "active", |
| 29 | + "observed_at": "2026-10-02T12:00:00Z", |
| 30 | + "reason_code": "seller_approved", |
| 31 | + }, |
| 32 | +) |
| 33 | +async with pool.connection() as conn: |
| 34 | + async with conn.transaction(): |
| 35 | + await conn.execute("UPDATE accounts SET status='active' WHERE id=%s", ("acct_1",)) |
| 36 | + await outbox.enqueue_prepared( |
| 37 | + conn, prepared, |
| 38 | + notification_type="account.status_changed", |
| 39 | + account_id="acct_1", |
| 40 | + caller_scope_id="seller-tenant-1", |
| 41 | + ) |
| 42 | + |
| 43 | +# Supervise this task in the application's worker lifecycle; cancel it on exit. |
| 44 | +await outbox.run_worker() |
| 45 | +``` |
| 46 | + |
| 47 | +Enqueue requires an open business transaction and performs no HTTP or credential |
| 48 | +resolution. A rollback publishes nothing. A commit persists the immutable body, |
| 49 | +key and headers for any later worker or process. `create_schema()` applies |
| 50 | +`adcp/notification_outbox.sql`, which ships in wheels and source distributions; |
| 51 | +production deployments can apply that additive DDL through their migration tool. |
| 52 | +There are no changes to task or reporting tables. Custom table names are supported. |
| 53 | + |
| 54 | +`caller_scope_id` must be the publisher's trusted caller/tenant identity, selected |
| 55 | +from server-side state. Never copy it from untrusted payload metadata. Account |
| 56 | +notifications also require the matching `account_id`. Caller-anchored principal |
| 57 | +and capabilities notifications use `account_id=None`; they need no fabricated |
| 58 | +account. Signing scope is an independently selected trusted key-service identity. |
| 59 | + |
| 60 | +The uniqueness scope is `(caller_scope_id, account_id, destination, |
| 61 | +notification_type, idempotency_key)`. The table indexes a SHA-256 digest of the |
| 62 | +first four fields so large destinations stay within PostgreSQL's index limits. |
| 63 | +Identical re-enqueue returns the original row ID without renewing the deadline. |
| 64 | +Different callers/accounts/destinations/types can use independent copies of a key. |
| 65 | +Reusing the same scoped key with different body, headers or signing scope fails. |
| 66 | +Use cryptographically random keys per fire, even though isolation permits reuse. |
| 67 | + |
| 68 | +AES-256-GCM encrypts the body and headers. The distinct `notification-outbox-v1` |
| 69 | +authenticated metadata binds type, destination, account, caller, signing scope, |
| 70 | +key, encryption-key ID and deduplication scope. Replay verifies the schema and |
| 71 | +body's key/type/account bindings again before resolving credentials or sending. |
| 72 | +Storage tampering fails authentication or binding validation and enters `invalid` |
| 73 | +without delivery. Known notification shapes are schema-validated at enqueue; |
| 74 | +extensions through `prepare_raw` must carry a matching `notification_type`. |
| 75 | + |
| 76 | +For legacy envelopes without `notification_type`, use these routing labels: |
| 77 | + |
| 78 | +| Preparation method | Outbox `notification_type` | |
| 79 | +| --- | --- | |
| 80 | +| `prepare_collection_list_changed` | `collection_list_changed` | |
| 81 | +| `prepare_property_list_changed` | `property_list_changed` | |
| 82 | +| `prepare_artifact_webhook` | `artifact-webhook` | |
| 83 | +| `prepare_revocation_notification` | `revocation-notification` | |
| 84 | + |
| 85 | +Workers claim one row with `FOR UPDATE SKIP LOCKED`, commit an expiring lease, |
| 86 | +and perform HTTP outside the transaction. A worker killed before acknowledgement |
| 87 | +leaves the row `in_flight`; another worker reclaims it after lease expiry. The |
| 88 | +worker bounds credential resolution plus HTTP by the lease duration and rechecks |
| 89 | +the lease and deadline immediately before HTTP. Default leases are 60 seconds; |
| 90 | +they must exceed the resolved sender's HTTP timeout by at least five seconds. |
| 91 | + |
| 92 | +Delivery is **at least once**: a receiver may accept a request just before its |
| 93 | +worker crashes. Every retry replays the exact bytes, headers and delivery key, |
| 94 | +with fresh signature metadata. Receivers must deduplicate by `idempotency_key` |
| 95 | +and a trusted authenticated publisher identity that survives signing-key rotation, |
| 96 | +and return 2xx for duplicates. Notification order is not guaranteed. Snapshot and |
| 97 | +change-feed reconciliation remain authoritative recovery paths. |
| 98 | + |
| 99 | +The retry horizon is 1–7 days **from enqueue**, including backlog time. Pending or |
| 100 | +abandoned work expires at that deadline. Connection failures, timeouts, transient |
| 101 | +DNS failures, temporary resolver failures, HTTP 408/425/429 and 5xx use exponential |
| 102 | +backoff with optional jitter from `RetryPolicy`. `max_attempts` is ignored for this |
| 103 | +durable path: the deadline governs retries. Other non-2xx responses, permanent URL |
| 104 | +or hook validation failures and permanently unknown signing scopes quarantine the |
| 105 | +row. Error columns contain bounded local discriminators, never peer bodies, |
| 106 | +credentials or resolver/hook diagnostics. |
| 107 | + |
| 108 | +`purge_expired()` expires overdue work and removes terminal rows seven days after |
| 109 | +their original deadline. `run_worker()` calls it periodically. This keeps duplicate |
| 110 | +bindings and delivery evidence during the retry window and a further retention |
| 111 | +window. Operational monitoring should query state, attempt count, deadline, |
| 112 | +`last_http_status` and `last_error`. Inspect quarantined rows through a trusted |
| 113 | +operator workflow; repair the cause before deliberately replaying with the |
| 114 | +original binding. Worker-loop errors are retried and cancellation leaves leases |
| 115 | +recoverable. |
| 116 | + |
| 117 | +Operator logs from `adcp.notification_outbox_pg` report the exception class and |
| 118 | +traceback frame locations (filename, function and line number) for delivery, |
| 119 | +unexpected resolver and worker-iteration failures. They exclude exception text |
| 120 | +and chains, frame locals and source-line snippets; raw `exc_info` is not attached. |
| 121 | +This identifies failure sites without logging credentials, URLs or peer content |
| 122 | +carried by exceptions. Cancellation propagates without a failure log. |
| 123 | + |
| 124 | +These diagnostics are separate from persisted errors. `last_error` remains a |
| 125 | +bounded class/status or quarantine discriminator, with no traceback metadata. |
| 126 | +For example, an unexpected resolver failure logs its original class and location |
| 127 | +before becoming the persisted `ScopeTransientlyUnavailable` retry marker. |
| 128 | +Iteration/database failures do not replace row errors; abandoned leases recover |
| 129 | +through the existing lease-expiry path. |
| 130 | + |
| 131 | +Use exactly one `sender` or `sender_resolver`. Both require RFC 9421 signing, |
| 132 | +SDK-owned IP-pinned transport, HTTPS destinations and private destinations disabled. |
| 133 | +The sender/resolver is application-owned; the outbox does not close it. Legacy |
| 134 | +HMAC/bearer one-shot senders can still prepare notifications, while this worker |
| 135 | +uses the protocol's default RFC 9421 transport. |
| 136 | + |
| 137 | +With `sender_resolver`, every enqueue requires `signing_scope_id`; fixed-sender |
| 138 | +outboxes require it to be absent. Each attempt calls the existing |
| 139 | +`WebhookSenderResolver.resolve(scope)` and validates its `WebhookSenderResolution` |
| 140 | +including advertised algorithms and timeout. Persisted scope is never changed or |
| 141 | +replaced by another tenant's default. Rotate signing keys by updating resolution |
| 142 | +under that same scope. `ScopeTransientlyUnavailable` retries; |
| 143 | +`ScopePermanentlyUnknown` quarantines. Unexpected resolver exceptions are treated |
| 144 | +as transient without persisting their messages. |
| 145 | + |
| 146 | +For encryption-key rotation, deploy workers with both old and new keys first, |
| 147 | +then select the new `encryption_key_id` and `encryption_key` for new rows. Supply |
| 148 | +old keys through `decryption_keys={"old-key-id": old_key}`. Existing rows retain |
| 149 | +their original key ID and remain readable and deduplicated. Retain each old key |
| 150 | +until all rows encrypted under it have been purged. Unknown key IDs or wrong key |
| 151 | +material quarantine rows; restoring a key does not automatically release them. |
0 commit comments