Skip to content

The 0.2.0b1 ledger upgrade breaks older seller pods still running during a rolling deploy, and their error names the wrong migration #1831

Open

Description

@bokelley

adcp-reporting-storage 0.2.0b1 upgrades a 0.1.0b4/0.2.0b0 ledger in place when the new release starts (#1787, thanks). #1787's operator note says this is a stopped-worker upgrade. Most sellers deploy with a rolling update, though. The Scope3 embedded sales agent runs 2 replicas, and every pod runs the reporting worker. So the first new pod upgrades the ledger while old pods are still serving and running their workers.

We drove that sequence with the published wheels against one PostgreSQL 16 database:

  1. adcp==9.3.1 + adcp-reporting-storage[pg]==0.1.0b4 (our production pair) seeds two generations with obligations, revisions and rows.
  2. adcp==10.0.0b8 + 0.2.0b1 runs create_schema(), which upgrades in place.
  3. The old pair does what a still-running old pod does.

Results for step 3:

Old pod action Result
create_schema() (pod restart) RaiseException: Reporting caller ownership requires a stopped-worker maintenance migration with HINT: Use migrate_legacy_reporting; retain legacy state in quarantine and backfill authoritative ownership.
read existing configurations, obligations, revisions, rows ok
close the next period of an existing generation (commit_obligation + commit_revision) LedgerConflictError: the ledger write violated an integrity rule
record a new generation (put_configuration) InvalidColumnReference: there is no unique or exclusion constraint matching the ON CONFLICT specification

The new pair still reads every seeded record afterwards, so nothing is corrupted. But for the length of the rollout, the old pods' reporting writes fail. An old pod that restarts in that window cannot start Reliable Reporting at all, and a rollback to the old release cannot boot against the upgraded ledger.

The restart error is the confusing part. It points the operator at migrate_legacy_reporting, the adcp 8 → 9 caller-ownership migration, which is the wrong fix and would quarantine state if followed.

Repro

tests/conformance/reporting/_published_ledger_seed.py plus one action run under the old pair after the upgrade:

# action "old_after_upgrade", run with adcp==9.3.1 + adcp-reporting-storage[pg]==0.1.0b4
await create_schema(pool, store, notifications=False)               # RaiseException (wrong hint)
await readback(store)                                                # ok
configuration = _configuration(ACCOUNTS[0])
obligation = await store.commit_obligation(_obligation(configuration, 2))
await store.commit_revision(*(await _async_revision(obligation, ACCOUNTS[0], 2)))   # LedgerConflictError
await store.put_configuration(_configuration("account-old-pod"))    # InvalidColumnReference (ON CONFLICT)

Sequence: seed (old pair) → upgrade (adcp==10.0.0b8 + 0.2.0b1) → old_after_upgrade (old pair) → read (new pair).

Suggested fix

Any one of these lets a rolling deploy go through:

  1. Expand/contract-safe upgrade. Keep a unique constraint the old writers' ON CONFLICT targets match (the four-column generation key, with offering_id defaulting to '') until a later contract release drops it. Old pods then keep writing account-level rows through the rollout, and the new release can still roll back.
  2. A clear refusal on the 9.x line. Publish an 0.1.0b5 (and a 9.x adcp floor) that recognizes a contract-2 ledger and refuses with "this ledger was upgraded by adcp-reporting-storage 0.2.x (storage contract 2); this release (contract 1) cannot write it", instead of the caller-ownership hint.
  3. Documentation, at minimum. Put the stopped-worker requirement, the symptoms above and the no-rollback consequence in MIGRATION_v9_to_v10.md and the storage README. Sellers can then stop old reporting workers (for example scale to one pod with the worker disabled) before the first new pod starts.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions