Repository navigation
Description
adcp-reporting-storage 0.2.0b1 upgrades a 0.1.0b4/0.2.0b0 ledger in place when the new release starts (#1787, thanks). #1787's operator note says this is a stopped-worker upgrade. Most sellers deploy with a rolling update, though. The Scope3 embedded sales agent runs 2 replicas, and every pod runs the reporting worker. So the first new pod upgrades the ledger while old pods are still serving and running their workers.
We drove that sequence with the published wheels against one PostgreSQL 16 database:
adcp==9.3.1+adcp-reporting-storage[pg]==0.1.0b4(our production pair) seeds two generations with obligations, revisions and rows.adcp==10.0.0b8+0.2.0b1runscreate_schema(), which upgrades in place.- The old pair does what a still-running old pod does.
Results for step 3:
| Old pod action | Result |
|---|---|
create_schema() (pod restart) |
RaiseException: Reporting caller ownership requires a stopped-worker maintenance migration with HINT: Use migrate_legacy_reporting; retain legacy state in quarantine and backfill authoritative ownership. |
| read existing configurations, obligations, revisions, rows | ok |
close the next period of an existing generation (commit_obligation + commit_revision) |
LedgerConflictError: the ledger write violated an integrity rule |
record a new generation (put_configuration) |
InvalidColumnReference: there is no unique or exclusion constraint matching the ON CONFLICT specification |
The new pair still reads every seeded record afterwards, so nothing is corrupted. But for the length of the rollout, the old pods' reporting writes fail. An old pod that restarts in that window cannot start Reliable Reporting at all, and a rollback to the old release cannot boot against the upgraded ledger.
The restart error is the confusing part. It points the operator at migrate_legacy_reporting, the adcp 8 → 9 caller-ownership migration, which is the wrong fix and would quarantine state if followed.
Repro
tests/conformance/reporting/_published_ledger_seed.py plus one action run under the old pair after the upgrade:
# action "old_after_upgrade", run with adcp==9.3.1 + adcp-reporting-storage[pg]==0.1.0b4
await create_schema(pool, store, notifications=False) # RaiseException (wrong hint)
await readback(store) # ok
configuration = _configuration(ACCOUNTS[0])
obligation = await store.commit_obligation(_obligation(configuration, 2))
await store.commit_revision(*(await _async_revision(obligation, ACCOUNTS[0], 2))) # LedgerConflictError
await store.put_configuration(_configuration("account-old-pod")) # InvalidColumnReference (ON CONFLICT)Sequence: seed (old pair) → upgrade (adcp==10.0.0b8 + 0.2.0b1) → old_after_upgrade (old pair) → read (new pair).
Suggested fix
Any one of these lets a rolling deploy go through:
- Expand/contract-safe upgrade. Keep a unique constraint the old writers'
ON CONFLICTtargets match (the four-column generation key, withoffering_iddefaulting to'') until a later contract release drops it. Old pods then keep writing account-level rows through the rollout, and the new release can still roll back. - A clear refusal on the 9.x line. Publish an
0.1.0b5(and a 9.x adcp floor) that recognizes a contract-2 ledger and refuses with "this ledger was upgraded by adcp-reporting-storage 0.2.x (storage contract 2); this release (contract 1) cannot write it", instead of the caller-ownership hint. - Documentation, at minimum. Put the stopped-worker requirement, the symptoms above and the no-rollback consequence in
MIGRATION_v9_to_v10.mdand the storage README. Sellers can then stop old reporting workers (for example scale to one pod with the worker disabled) before the first new pod starts.
Activity