fix(reporting-storage): upgrade 0.1.0b4/0.2.0b0 ledgers in place and pin the adcp storage contract (#1763)

MIGRATION_v9_to_v10.md

@@ -53,3 +53,24 @@ the SDK-relevant inventory is [docs/adcp-3.3-sdk-inventory.md](docs/adcp-3.3-sdk
 | Generated enums name each value once | The `_1` alias members codegen emitted when an `anyOf` arm repeated an enum value (`System.zip_1`, `Field17.ad_request_url_1`, 19 enums on 10.0.0b4) are gone; each value has one member, the first-declared one. Code that referenced an alias name (`System.zip_1`) must use the real member (`System.zip`); `System("zip")` and `member.value` are unchanged. |
 | `list_account_changes` pages are checked and projected per caller | AdCP 3.3 requires non-empty `changed_paths` on every `adcp` / `seller_operator` change record (`[""]` for a whole-resource create, delete or purge) and lets the originating request's `idempotency_key` reach only the principal that sent it. A handler on 3.3 or later that returns such a record without `changed_paths` now fails the call with `VALIDATION_ERROR` (`field` `/changes/N/changed_paths`), whatever the validation mode; and every record loses `idempotency_key` unless it is an `adcp` record built with `adcp.account_change_feed.attribute_account_change(record, originator)` whose originator is the caller (`ToolContext.tenant_id` + `caller_identity`). Persist the originator when the mutation commits and attribute records when serving the page. `WebhookSender.prepare_account_change_recorded` / `send_account_change_recorded` also accept the change record itself (with `subscriber_id=`), refuse one without `changed_paths`, and never carry its `idempotency_key`. |
 | `get_adcp_capabilities` reports `adcp.implementation` | `serve()` (and `adcp.decisioning.serve()`) now adds AdCP 3.3 `adcp.implementation.sdks[]` to capabilities served at 3.3 or later: one entry `{"name": "adcp", "version": <installed adcp version>, "components": [...]}`, listing only components the server provably wires agent-wide (`transport` always; `types` for Decisioning platforms; `schema_validation` with strict request and response validation and no `2.5` in `supported_versions`; `authentication` with `auth=BearerTokenAuth(...)` and no widened discovery set; `idempotency` when every advertised mutating task is `@IdempotencyStore.wrap`ped; `webhooks` / `signing` only when the SDK owns and signs every outbound webhook; `lifecycle` never automatically). A 3.0 to 3.2 caller never sees it. Tests that compare a whole 3.3 capabilities response need the new key. Opt out with `serve(report_implementation=False)` (or `ServeConfig(report_implementation=False)`); add components or other SDKs with `ImplementationReport(extra_components=[...], extra_sdks=[...])`; or declare the block yourself (`Adcp(implementation=...)` on `DecisioningCapabilities`, `capabilities_response(implementation=...)`, or in your own handler's response), which the SDK never overwrites. `adcp.server.adcp_sdk_entry()` builds this SDK's entry for a hand-written block. `DecisioningCapabilities.adcp.implementation` was previously dropped from the projection; it is now emitted. |
+
+## Reporting storage contract (adcp-reporting-storage)
+
+adcp 10.0.0b4 widened the reporting ledger's generation key with `offering_id`
+(AdCP 3.3 reporting subscriptions). The layout `adcp` and
+`adcp-reporting-storage` share is now named by one integer on each side,
+`adcp.reporting.ledger.storage_contract.REPORTING_STORAGE_CONTRACT` and
+`adcp_reporting_storage.STORAGE_CONTRACT` (2 for SDK 10; SDK 9 and
+adcp 10.0.0b0–b3 are 1):
+
+- `adcp-reporting-storage` 0.2.0b1 requires `adcp>=10.0.0b4,<11`; the
+  0.2.0b0 / 0.1.0b4 releases do not work with 10.0.0b4 or later
+  ([#1763](https://github.com/adcontextprotocol/adcp-client-python/issues/1763)).
+- Startup (`ReliableReportingService`, `postgres_storage`) refuses a pair whose
+  contracts differ with `REPORTING_SCHEMA_BEHIND`, naming both versions and
+  the release to install.
+- An existing SDK 9 ledger is upgraded in place by `create_schema()` or
+  `python -m adcp migrate`: `offering_id` is added with `''` for every
+  existing account-level generation, obligation and status, and the applied
+  schema versions are recorded. Stop every SDK 9 reporting worker first; see
+  [docs/reporting-ledger-migration.md](docs/reporting-ledger-migration.md).

docs/reliable-reporting-service.md

@@ -412,6 +412,10 @@ platform = reporting.install(MySeller())  # then serve the platform as usual
     (`--dsn` defaults to `$ADCP_DATABASE_URL`, then `$DATABASE_URL`). Startup
     never changes the schema: it refuses one that is behind with
     `REPORTING_SCHEMA_BEHIND`, naming the command and the pending versions.
+    Startup also refuses an installed `adcp-reporting-storage` whose storage
+    contract differs from this adcp's (`adcp.reporting.ledger.storage_contract`),
+    with the same code, naming both versions and the release to install, so an
+    incompatible pair fails once at startup rather than on every turn (#1763).
   - `0002 reporting_subscription_generations` widens the generation key and
     `0006 reporting_status_checkpoint_offerings` keys status checkpoints and
     status events by the generation's resolved offering. Both are

docs/reporting-ledger-migration.md

@@ -281,3 +281,38 @@ history, not to authorize an unfrozen new publication. High-level users of
 `ProducerOfferings(currency=...)` keep their fixed-currency convenience; new
 obligations freeze that value. See [multi-account currency resolution and
 monetary validation](reporting-currency.md) for the resolver and type examples.
+
+## Upgrading a ledger created by adcp-reporting-storage 0.1.0b4 or 0.2.0b0 (#1763)
+
+adcp 10.0.0b4 widened the configuration generation key with `offering_id`
+(AdCP 3.3 reporting subscriptions) and reads it from `reporting_configurations`,
+`reporting_obligations` and `reporting_consumer_statuses`. The storage releases
+before 0.2.0b1 do not create the column, so that pair fails every reporting
+turn with `column "offering_id" does not exist`. From 0.2.0b1 the pair is
+refused at startup instead (storage contract 1 versus 2; see
+`packages/adcp-reporting-storage/README.md`), and both schema entry points
+upgrade an existing ledger in place:
+
+- `PgReportingLedgerStore.create_schema()`, which `ReliableReportingService`
+  runs at startup when built with `store=`;
+- `python -m adcp migrate`, which `postgres_storage(...)` requires to have run.
+
+The upgrade (migration `0002`) adds `offering_id TEXT NOT NULL DEFAULT ''` to
+the widened tables, so every existing account-level generation, obligation and
+status keeps its identity with `offering_id = ''`; rebuilds the generation
+primary key and the obligation period index with the new column; and re-points
+the managed-delivery foreign keys. No evidence row is rewritten or dropped. The
+applied versions are recorded in `adcp_schema_migrations` (`create_schema()`
+applies the ledger's own `0001`–`0003`; `python -m adcp migrate` also applies
+the later ones, which only `postgres_storage` and the notification stores
+need), re-running is a no-op,
+and concurrent boots serialize on the reporting schema advisory lock inside
+one transaction. It is a stopped-worker upgrade: stop every worker of the
+earlier release first, because their configuration writes name the
+four-column key.
+
+`tests/conformance/reporting/test_reporting_storage_published_upgrade_pg.py`
+seeds ledgers with the actual published wheel pairs (`adcp==9.3.1` with
+`adcp-reporting-storage==0.1.0b4`, `adcp==10.0.0b2` with `0.2.0b0`) and proves
+the retained evidence reads back unchanged, a restart rebinds the stored
+generations, and the upgraded catalog is identical to a fresh migration.

docs/reporting-subscriptions-errors.md

@@ -16,7 +16,7 @@ Every SDK error carries `code` and `recovery` (`transient`, `correctable` or `te
 | `COVERAGE_INPUT_INCOMPLETE`, `COVERAGE_INPUT_INVALID` | `CoverageInputError` | n/a | correctable | supply the named input |
 | `COVERAGE_BUDGET_EXCEEDED` | `CoverageInputError` | n/a | correctable | narrow the input |
 | `REPORTING_STORAGE_NOT_DURABLE` | `ReliableReportingConfigurationError` | n/a | correctable | `storage=postgres_storage(...)`, or `ADCP_ENV=development` |
-| `REPORTING_SCHEMA_BEHIND` | `ReliableReportingConfigurationError` | n/a | correctable | run `python -m adcp migrate` |
+| `REPORTING_SCHEMA_BEHIND` | `ReliableReportingConfigurationError` | n/a | correctable | run `python -m adcp migrate`; or, when the installed `adcp-reporting-storage` implements another storage contract than this `adcp` ([#1763](https://github.com/adcontextprotocol/adcp-client-python/issues/1763)), install the release the message names |
 | `SUBSCRIPTIONS_REQUIRE_STANDARD_DIMENSIONS` | `ReliableReportingConfigurationError` | n/a | correctable | publish `standard_dimensions` on the named offering |
 | `SUBSCRIPTION_ROSTER_UNAVAILABLE` | `ReliableReportingConfigurationError` | n/a | correctable | serve `list_accounts`, or supply `SubscriptionOptions.accounts` |
 

docs/reporting-subscriptions-seller.md

@@ -103,7 +103,7 @@ account.authorization = wind_down_grant(account.authorization)
 
 Pass one storage object: `memory_storage()` in development, `postgres_storage(dsn=... | pool=...)` in production. Process-local storage is refused outside development (`REPORTING_STORAGE_NOT_DURABLE`).
 
-Apply schema changes with `python -m adcp migrate [--status | --dry-run]` before starting a release. Startup refuses a schema that is behind (`REPORTING_SCHEMA_BEHIND`). `0002 reporting_subscription_generations` and `0006 reporting_status_checkpoint_offerings` are stopped-worker upgrades: stop every reporting worker first.
+Apply schema changes with `python -m adcp migrate [--status | --dry-run]` before starting a release. Startup refuses a schema that is behind (`REPORTING_SCHEMA_BEHIND`), and an installed `adcp-reporting-storage` of another storage contract than this `adcp` (same code; the message names the release to install). `0002 reporting_subscription_generations` and `0006 reporting_status_checkpoint_offerings` are stopped-worker upgrades: stop every reporting worker first.
 
 ## Workers and scaling
 

packages/adcp-reporting-storage/README.md

@@ -53,10 +53,40 @@ client you build; the extras only pin the client libraries.
 
 This package implements the protocols of one `adcp` major and imports a few
 private helpers of `adcp.reporting` to do so (`adcp.reporting.ledger._record_rows`,
-`_delivery_state`, `status_snapshot`). Its dependency range is therefore one
-major (`adcp>=9.0.0,<10`), and a new `adcp` major means a new release here.
+`_delivery_state`, `status_snapshot`). The column lists those helpers read are
+the **storage contract**, numbered on both sides
+(`adcp_reporting_storage.STORAGE_CONTRACT`,
+`adcp.reporting.ledger.storage_contract.REPORTING_STORAGE_CONTRACT`) and bumped
+together whenever the shared layout changes. The dependency floor is the first
+`adcp` release of the contract this package implements, so an incompatible pair
+does not resolve; if one is forced, building a PostgreSQL store refuses at
+startup, naming both versions and the release to install.
+
+| adcp-reporting-storage | storage contract | adcp |
+|---|---|---|
+| 0.1.0b4 (SDK 9 line) | 1 | `>=9.0.0,<10` |
+| 0.2.0b0 | 1 | `>=9.0.0,<11` (**do not pair with 10.0.0b4+**, [#1763](https://github.com/adcontextprotocol/adcp-client-python/issues/1763)) |
+| 0.2.0b1+ | 2 | `>=10.0.0b4,<11` |
+
 `adcp` never imports this package; a test in the main repository enforces that.
 
+### Upgrading a ledger created by 0.1.0b4 or 0.2.0b0
+
+Both `PgReportingLedgerStore.create_schema()` (what `ReliableReportingService`
+runs at startup when given `store=`) and `python -m adcp migrate` upgrade an
+existing ledger in place: `offering_id` is added to `reporting_configurations`,
+`reporting_obligations`, `reporting_consumer_statuses` and the managed-delivery
+tables, backfilled with `''` for every existing account-level generation, the
+generation keys and unique indexes are widened, and the applied schema
+versions are recorded in `adcp_schema_migrations` (`create_schema()` applies
+and records the ledger's own `0001`–`0003`; `python -m adcp migrate` also
+applies the later ones, which only `postgres_storage` and the notification
+stores need). No evidence row is
+rewritten or dropped. The upgrade is transactional, serialized under the reporting schema
+advisory lock, and a no-op when re-run. It is a **stopped-worker** upgrade:
+stop every reporting worker of the earlier release first (its configuration
+writes name the four-column key), run the upgrade, then start the new release.
+
 ## Versioning
 
 0.x: breaking changes may land in a minor release, each with a migration note in

packages/adcp-reporting-storage/pyproject.toml

@@ -33,11 +33,13 @@ dependencies = [
     # ``adcp.reporting`` helpers, so the range is one major: a new ``adcp``
     # major means a new release here.
     #
-    # On next this range also admits adcp 10 pre-releases: next releases this
-    # distribution as its SDK 10 companion (0.2.0-beta.N), and the workspace
-    # must still install while pyproject.toml carries main's version. At
-    # ratification the range narrows to ``adcp>=10.0.0,<11`` (RELEASING.md).
-    "adcp>=9.0.0,<11",
+    # The floor is the first adcp release whose storage contract this package
+    # implements (``adcp_reporting_storage.contract``): 10.0.0b4 added
+    # ``offering_id`` to the ledger column lists, which 0.2.0b0's DDL lacked, so
+    # the pair installed cleanly and failed every reporting turn (#1763). Move
+    # the floor together with STORAGE_CONTRACT. At ratification the range
+    # narrows to ``adcp>=10.0.0,<11`` (RELEASING.md).
+    "adcp>=10.0.0b4,<11",
     # RFC 8785 canonical JSON for row bodies and object keys; the same pin as adcp.
     "rfc8785>=0.1.4",
 ]

packages/adcp-reporting-storage/src/adcp_reporting_storage/__init__.py

@@ -35,6 +35,10 @@
 
 if TYPE_CHECKING:
     __version__: str
+    from adcp_reporting_storage.contract import STORAGE_CONTRACT as STORAGE_CONTRACT
+    from adcp_reporting_storage.contract import (
+        require_adcp_contract as require_adcp_contract,
+    )
     from adcp_reporting_storage.feed.pg import PgReportingFeedStore as PgReportingFeedStore
     from adcp_reporting_storage.inline_storage import (
         PgReportingSealStore as PgReportingSealStore,
@@ -214,6 +218,9 @@
 
 #: Public name -> (module under this package, attribute). Resolved on first use.
 _LAZY_EXPORTS: dict[str, tuple[str, str]] = {
+    # The storage contract shared with adcp (see ``contract``)
+    "STORAGE_CONTRACT": ("contract", "STORAGE_CONTRACT"),
+    "require_adcp_contract": ("contract", "require_adcp_contract"),
     # Stores
     "PgReportingLedgerStore": ("ledger.pg", "PgReportingLedgerStore"),
     "PgReportingReconciliationStore": ("ledger.delivery_pg", "PgReportingReconciliationStore"),
@@ -320,6 +327,7 @@
 
 _LAZY_SUBMODULES = frozenset(
     {
+        "contract",
         "feed",
         "inline_storage",
         "ledger",

packages/adcp-reporting-storage/src/adcp_reporting_storage/contract.py

@@ -0,0 +1,93 @@
+"""The storage contract this distribution implements, checked against the installed ``adcp``.
+
+``adcp`` reads and writes the reporting ledger through private column lists
+(``adcp.reporting.ledger._record_rows``) and this distribution ships the DDL
+those lists are read against. The two must agree exactly: a column one side
+knows and the other does not fails every reporting turn with
+``UndefinedColumn`` (adcp-client-python#1763). The agreement is named by one
+integer on each side, bumped together whenever the shared layout changes:
+
+* **1** — the SDK 9 layout and adcp 10.0.0b0–b3: a configuration generation is
+  ``(account_id, consumer_id, delivery_config_id, delivery_config_version)``.
+  Shipped by adcp-reporting-storage 0.1.0b4 and 0.2.0b0.
+* **2** — adcp 10.0.0b4 and later: ``offering_id`` on configurations,
+  obligations and consumer statuses (AdCP 3.3 reporting subscriptions; schema
+  migration ``0002``). Shipped by adcp-reporting-storage 0.2.0b1 and later.
+
+:func:`require_adcp_contract` runs when a PostgreSQL ledger store is built, so
+an incompatible pair fails at startup with the pair named and the fix spelled
+out, instead of on the first ledger read.
+"""
+
+from __future__ import annotations
+
+from importlib import import_module
+from importlib.metadata import PackageNotFoundError, version
+
+__all__ = ["STORAGE_CONTRACT", "adcp_storage_contract", "require_adcp_contract"]
+
+#: The reporting storage contract this distribution's DDL and stores implement.
+STORAGE_CONTRACT = 2
+
+
+def adcp_storage_contract() -> int:
+    """The storage contract the installed ``adcp`` reads the ledger with.
+
+    adcp declares it from 10.0.0b6 (``adcp.reporting.ledger.storage_contract``).
+    Earlier releases are classified by the column list that distinguishes the
+    contracts: 10.0.0b4 and b5 select ``offering_id`` (contract 2); everything
+    before reads the four-column generation (contract 1).
+    """
+    try:
+        declared = import_module("adcp.reporting.ledger.storage_contract")
+    except ImportError:
+        from adcp.reporting.ledger import _record_rows
+
+        columns = [item.strip() for item in _record_rows._OBLIGATION_COLUMNS.split(",")]
+        return 2 if "offering_id" in columns else 1
+    return int(declared.REPORTING_STORAGE_CONTRACT)
+
+
+def _installed(distribution: str) -> str:
+    try:
+        return version(distribution)
+    except PackageNotFoundError:
+        return "unknown"
+
+
+def require_adcp_contract() -> None:
+    """Refuse an ``adcp`` whose storage contract differs from this distribution's.
+
+    Raises ``adcp.reporting.ReliableReportingConfigurationError`` (code
+    ``REPORTING_SCHEMA_BEHIND``) naming both installed versions, both contracts
+    and the release to install.
+    """
+    expected = adcp_storage_contract()
+    if expected == STORAGE_CONTRACT:
+        return
+    from adcp.reporting import ReliableReportingConfigurationError
+
+    adcp_version = _installed("adcp")
+    storage_version = _installed("adcp-reporting-storage")
+    if expected > STORAGE_CONTRACT:
+        fix = (
+            f"Install the adcp-reporting-storage release that implements contract {expected} "
+            f"(newer than {storage_version}), or adcp 10.0.0b4–10.0.0b5 with this one."
+        )
+    else:
+        fix = (
+            f"Install adcp 10.0.0b4 or later (storage contract {STORAGE_CONTRACT}), or "
+            "adcp-reporting-storage 0.2.0b0 with this adcp."
+        )
+    message = (
+        f"adcp {adcp_version} reads the reporting ledger with storage contract {expected}, "
+        f"but adcp-reporting-storage {storage_version} implements contract {STORAGE_CONTRACT}; "
+        f"the pair would fail every reporting turn. {fix}"
+    )
+    try:
+        error = ReliableReportingConfigurationError(
+            message, code="REPORTING_SCHEMA_BEHIND", config_field="adcp-reporting-storage"
+        )
+    except (TypeError, ValueError):  # an adcp that predates the code or the keywords
+        error = ReliableReportingConfigurationError(message)
+    raise error