diff --git a/docs/design-history.md b/docs/design-history.md index 2b62ed63..c291a577 100644 --- a/docs/design-history.md +++ b/docs/design-history.md @@ -2965,3 +2965,190 @@ v1 for deployments that verify that way, and a publisher that pushes its own cat its own key still can, because the push path accepts the domain's own publisher as well as any contributor its manifest names. What changed is which path the publisher guide describes: the guide follows the console, not the other way round. + +## A refusal carries its answer's header fields, and only a refusal does + +A peer that refuses a call often says in its header fields what it wants next. A Web Bot +Auth verifier names the signature it requires in `Accept-Signature` (WG-00 §5.3), a +draining gateway sets `Retry-After`, and a delivery edge's 403 names the publisher's terms +and Exchange in `X-Content-Rules` and `X-FORA-Exchange`. None of the three clients handed +those fields back. A conformance suite checking a verifier's `Accept-Signature` had to post +raw HTTP and parse the raw answer, which is the work these clients exist to do for it. + +Each client's failure now carries them: `CallError.headers` in Python, `ForaCallError.headers` +in TypeScript, and `CallError.Header` in Go, with `resolvers.FetchError.Header` beneath it on +the content leg. + +**Only an error answer carries fields.** Status 400 or above, on either leg. A failure where +no answer arrived has none to carry, and a 2xx the client could not read is not a refusal. +A redirect is left out on purpose. Its `Location` names a URL the peer chose, credentials +included, and the SDK already withholds that target everywhere else: a signed leg refuses +the redirect and redacts its target (see "A signed leg refuses redirects; the guarded fetch +follows them"), and a key directory's refused redirect names only the address read (see "A +failed document read says when the answer was a 404 or a redirect"). Carrying the fields +would hand the target straight back. The rule has one owner per language: the failure's +constructor in Python and TypeScript, so every site passes the answer's fields and the +constructor decides; and `failure.RefusalHeader` in Go, which both legs call. + +**The fields are values, never prose.** They are the peer's, unvalidated, and no client +renders them into an error message or a log line. Like `peer_message`, they are bounded +only by the HTTP client's own header limit, and truncating them is the business of +whoever displays them. + +**One reading across three HTTP libraries.** net/http canonicalizes field names and keeps +each line of a repeated field; undici lowercases names and decodes values as UTF-8; httpx +keeps names as sent. A caller reads a field by any spelling of its name, and a repeated +field as one value joined with ", " (RFC 9110 §5.3). Go keeps the lines in its +`http.Header`, and `strings.Join(Header.Values(name), ", ")` reads the same value. In +TypeScript a value the Fetch `Headers` type cannot hold, a character above U+00FF, is +left out rather than thrown on, because describing a failure must not become a second +failure. `refusal-header-vectors.json` pins the reading in all three SDKs, captured from the +real Go client. It records each row's own field lines only, since `Date` and +`Content-Length` change with the run and say nothing about the rule. + +**Go records the answer itself.** connect-go's `Error.Meta()` holds response headers only +when the body decoded as a Connect envelope, so a gateway's plain-text 503 and its +`Retry-After` would have carried nothing. The client's transport already recorded that a +response arrived, to tell a peer's Connect code from one connect-go derived; it now keeps +that answer's status and copied fields too. The same record fixes `CallError.Status` on the +RPC path, which its doc described and nothing set. + +**The record is one request's.** An interceptor passed to `WithInterceptors` runs inside the +client's own and may call `next` again to retry a call, so one call can send several +requests. Each request clears the record, and the strict-decoding capture of its error body, +before it is sent. A failure therefore describes the last request the call sent: a transport +failure after an earlier error answer carries no code, status or header fields, and an +earlier attempt's error envelope is never judged in its place. The record is cleared before +the request rather than when no answer comes back, because a 2xx, an error body past the +read cap and one in a coding the client did not ask for all arrive and store nothing. The +record cannot tell which attempt an error came from, so an interceptor that builds its own +error, returns an earlier attempt's or sends attempts concurrently gets the last request's +record beside it: a retrying interceptor returns its last attempt's error and sends its +attempts one at a time. Tying every error to its attempt would mean threading an attempt +through each layer, and a strict-decoding refusal does not even wrap the Connect error it +judged. The Python and TypeScript clients build each failure from the response their send +returned, so a retry cannot mix two attempts there. + +**A refused redirect's target stays out of Go's RPC errors, and a test now says so.** +net/http wraps the refusal in a `*url.Error` naming the raw `Location`, and the Go client +keeps that out of its errors only because connect-go discards the wrapper before the error +surfaces. A test now pins that on every RPC face, so a dependency that stopped discarding it +fails a test rather than leaking into a log. + +**Differences left as they are.** TypeScript's RPC read cap refuses without a status, so +that failure carries no fields, where Python's carries both. Go reports `Status` 0 for a +refused redirect and for a 2xx it could not read, where the two JSON clients report the +status. A 2xx that is not a FORA answer is a refusal in Go, which connect-go classifies by +its content type, and malformed in Python and TypeScript. Each predates this change, and +none decides whether fields are carried. + +## A signer the protocol forbids lives in a test package + +fora.proto requires an acceptance to name its requester: "A signer refuses to sign bytes +that name an empty requester, and a verifier refuses an acceptance whose canonical bytes +name one, even when the signature over those bytes verifies." The SDK's acceptance signers +are that signer in all three languages, so none of them can produce the input a verifier's +refusal is tested with. The shared corpora recorded those bytes in their `refused` lists +("what a signer without the check would produce"), but only the Go vector emitter could make +them, through an unexported renderer, and a conformance suite had to write the canonical +bytes and the Ed25519 signature by hand. + +The SDK now ships that signer, as test support. Each language has twins of its four +acceptance functions that skip the empty-requester refusal and nothing else: Go's +`sdk/go/helperstest`, Python's `fora_sdk.testing`, and TypeScript's +`@fora-protocol/sdk/testing`. + +**Not a flag on the protocol's signer.** A parameter that turns the refusal off would leave +every production call one argument away from signing what the protocol forbids. That is the +shape this file already rules out for the SSRF guard ("An option that could silently disarm +the guard is exactly how a security property becomes advisory"), and the reasoning carries +over unchanged. + +**Not a client seam either.** The pre-signing hook and raw mode shape the message a +conforming signer then signs. Neither makes the SDK produce a signature the protocol says a +signer refuses to make. An artifact a conforming party must refuse to produce is a different +kind of thing, so it lives where production code does not reach. In Go that is a test +package, named the way `net/http/httptest` is. In Python it is a module the package root +does not re-export, and in TypeScript an export path of its own. A test in each language +fails if production code imports it. + +**Only the requester check is skipped.** It is the refusal the `refused` vectors record, so +what the twins produce is pinned byte for byte in all three languages. The other refusals +stay. Skipping them would reach inputs the shared vectors do not pin, such as an empty item +list or an empty item field. Python and TypeScript build those payloads by hand, and their +omission of an empty member is pinned only for the shapes the vectors carry. + +**One renderer per language serves both.** The canonical renderer takes the requester check +as a parameter and runs it where the protocol's signer does. The protocol's signers pass +their check, and the test package passes an unexported one that admits any requester, so +nothing outside it can name the bypass. In Go the renderer and the other checks moved into +`sdk/go/internal/canonicalsign`. In Python they are private functions of `fora_sdk.core`, and +in TypeScript a module neither package manifest exports. Tests pin the text and the order of +every refusal; they passed before the move and after it. + +**The Go purity guards follow the code.** The IO-leaf guard and the connect-free guards of +helpers and core used to scan one directory each, so the move took the renderer out of their +reach. They now scan every package of the module their package is built from, through other +packages too, and helpers' guards pin the canonical renderer among them, so code that moves +into an internal package stays guarded. The walk reads import specs with `go/parser` and +runs no go command. + +The sentence in fora.proto is unchanged, because it describes a conforming signer and the +twins are not one: an acceptance they sign is one a conforming party must never send. + +## The Python client can refuse a request the strict contract does not describe + +A Python verb takes a plain dict as well as the generated request model, and the models +ignore a member they do not declare (`extra="ignore"` in `gen/python/wire/base.py`, the +forward-compatible choice for reading an answer). So a misspelled key passes the default +`validation="strict"` check, which is the generated model's field-level one, and is signed +and sent. A Connect server's JSON codec discards the unknown field, and the request is +served as if the key were absent. A harness whose negative test mutates a key then has a +test that passes while testing nothing. + +`ClientConfig.strict_requests` closes that with two checks, and is off by default. + +**A type gate where every message enters.** Every verb that takes a dict or a model copies +it once, before it stamps the envelope, and that copy is where the switch refuses anything +but the verb's own generated model. It runs before the verb resolves an endpoint, reads a +manifest or signs, so a refused message costs nothing. A model of another message is +refused there too, and the error names the model to pass. + +**The strict check on the stamped message.** The gate cannot see what the client adds +itself: `ver`, the configured `requester`, the offers a purchase reflects, and the items +and acceptances it builds. So the message as stamped is checked just before it is signed, +against the published strict JSON Schema of the request message and the cross-field rules, +by the same check strict decoding runs on an answer. The generated model's own check does +not run the cross-field rules, so a catalog push whose term carries no price is now refused +locally rather than by the server. + +**Not a value of `validation`.** `Validation` is mapped across the three SDKs with the same +two values, and the switch has no counterpart in Go or TypeScript. A Go verb takes the +generated message, which has no member the proto does not declare, and +`WithValidation(ValidationStrict)` runs its rules, cross-field ones included. A TypeScript +verb is typed with the request input types, so a misspelled key in an object literal does +not compile. A third Python value would make the mapped type diverge. The switch is +therefore a field of its own, independent of `validation`: turning `validation` off leaves +it in force. + +**Off by default.** A dict is the common way to build a request. A purchase also reflects +each offer exactly as discovery returned it, so an offer from a newer minor version carries +a member this SDK does not know, and under the switch that purchase is refused. That is +right for a harness and wrong as a default. + +**`RawBody` is exempt.** Raw mode exists to put on the wire a message the SDK would never +build, so the switch checks nothing about it and adds nothing to it. + +**The refusal names the member.** A harness has to see which key was refused, so the +refusal carries the text strict decoding gives an answer: the path and the schema's +complaint, which quotes the member's name and, when a field breaks a rule, its value. A +reflected offer's member is the Exchange's text, and a strict client already gets the same +text when the discovery answer carrying that offer is refused. It is bounded by the read +cap that answer was read under, and quoted with escapes, so it cannot break a log line. + +**The gap left.** A keyword argument that a generated model's constructor does not declare +is dropped by the model before the SDK sees it, so neither check can report it. A type +checker does: the models are dataclass transforms, and mypy reports the unexpected keyword +argument. A model built with `model_construct` skips validation and still drops an +undeclared keyword, but it renders a nested value exactly as given. The check of the stamped +message therefore refuses an unknown member inside such a value. diff --git a/docs/sdk-parity-matrix.md b/docs/sdk-parity-matrix.md index 5d640e48..9479f4f6 100644 --- a/docs/sdk-parity-matrix.md +++ b/docs/sdk-parity-matrix.md @@ -10,9 +10,9 @@ Regenerate: python3 scripts/gen-parity-matrix.py Drift-gated in scripts/ci-local.sh. Narrative/rationale: docs/design-history.md. --> -Go is the oracle (`sdk/go/{helpers,resolvers,core,connect,connectserver,profiles/music}`); Python and TS mirror it. This document is **generated** from the same two artifacts CI already enforces against the code, so it cannot drift from the real surface — a mismatch fails the API-surface gate or the corpus-completeness gate before it can reach this file. +Go is the oracle (`sdk/go/{helpers,resolvers,core,connect,connectserver,profiles/music,helperstest}`); Python and TS mirror it. This document is **generated** from the same two artifacts CI already enforces against the code, so it cannot drift from the real surface — a mismatch fails the API-surface gate or the corpus-completeness gate before it can reach this file. -**At a glance:** 192 symbols at cross-language parity · 17 documented divergences · 229 Go-idiomatic exclusions · 45 conformance corpora, each tri-replayed. +**At a glance:** 196 symbols at cross-language parity · 17 documented divergences · 229 Go-idiomatic exclusions · 46 conformance corpora, each tri-replayed. Layering (L1 pure trust core vs L2 I/O resolvers), the SSRF transport-wiring invariant, and naming conventions are recorded in [`design-history.md`](./design-history.md). @@ -242,6 +242,15 @@ Legend: a name = the public face in that language · `—` = intentionally none | `Music` | `Music` | `Music` | | `ProfileID` | `PROFILE_ID` | `ProfileID` | +### helperstest — test support: acceptance signers that admit an empty requester + +| Go | python | ts | +|---|---|---| +| `CanonicalAcceptanceBytesAnyRequester` | `jcs_acceptance_payload_any_requester` | `acceptancePayloadAnyRequester` | +| `CanonicalRequestAcceptanceBytesAnyRequester` | `jcs_request_acceptance_payload_any_requester` | `requestAcceptancePayloadAnyRequester` | +| `SignOfferAcceptanceAnyRequester` | `sign_offer_acceptance_any_requester` | `signOfferAcceptanceAnyRequester` | +| `SignRequestAcceptanceAnyRequester` | `sign_request_acceptance_any_requester` | `signRequestAcceptanceAnyRequester` | + ## Documented divergences & DECISIONs Deliberate, reason-backed asymmetries. The allowlist is **shrink-only** — a new undocumented gap cannot ship green. Each entry's rationale is the `allowlist_reason` from the symbol map; DECISION anchors are cross-checked by `test_api_surface_parity.py`. @@ -557,6 +566,7 @@ Go emits each `*-vectors.json` oracle; Python and TS replay it. The completeness | `connect/testdata/client-request-vectors.json` | ✅ | ✅ | ✅ | | `connect/testdata/connect-error-vectors.json` | ✅ | ✅ | ✅ | | `connect/testdata/error-detail-wire-vectors.json` | ✅ | ✅ | ✅ | +| `connect/testdata/refusal-header-vectors.json` | ✅ | ✅ | ✅ | | `connect/testdata/synthesized-detail-vectors.json` | ✅ | ✅ | ✅ | | `connect/testdata/transport-failure-vectors.json` | ✅ | ✅ | ✅ | diff --git a/package.json b/package.json index 36bb7a3e..a2a831ac 100644 --- a/package.json +++ b/package.json @@ -40,6 +40,7 @@ "./identity": "./sdk/ts/core/identity.ts", "./hono": "./sdk/ts/hono/middleware.ts", "./acceptance": "./sdk/ts/src/acceptance.ts", + "./testing": "./sdk/ts/src/testing.ts", "./resolvers": "./sdk/ts/resolvers/index.ts", "./resolvers/edge": "./sdk/ts/resolvers/edge.ts", "./client": "./sdk/ts/client/index.ts", diff --git a/proto/CHANGELOG.md b/proto/CHANGELOG.md index 91c7c42c..cc3d44f5 100644 --- a/proto/CHANGELOG.md +++ b/proto/CHANGELOG.md @@ -1,5 +1,85 @@ # FORA Protocol Changelog +## Unreleased + +### Breaking against v1.1.0 + +**SDK users** + +- **Go client errors.** `connect.CallError` and `resolvers.FetchError` gain a `Header` + field, an `http.Header`, so neither struct compares with `==` any more. An RPC + `CallError` now carries the `Status` of the peer's error answer, which v1.1.0 left at 0, + so its message names that status, as the content leg's always did. + +### Changes since v1.1.0 + +**Three more SDK capabilities for conformance and e2e harnesses (no wire change).** A +harness can now read the header fields a refusal arrived with, sign an acceptance that +names an empty requester to prove that a receiver refuses it, and, in Python, have the +client refuse a request the contract does not describe before it is signed. + +- **Refusal headers.** A refusal carries the header fields of the answer that refused: + `CallError.headers` (Python, a mapping looked up case-insensitively), + `ForaCallError.headers` (TypeScript, a Fetch `Headers`) and `CallError.Header` with + `resolvers.FetchError.Header` (Go, an `http.Header`), on the RPC legs and the delivery + fetch alike. A Web Bot Auth verifier's `Accept-Signature`, a gateway's `Retry-After` and + an edge's discovery headers are read through the SDK; `parse_discovery_hint`, + `parseDiscoveryHint` and `helpers.ParseDiscoveryHint` read the last with the error's + status. Only an error answer, status 400 or above, carries fields: a local failure, a + transport failure, a 2xx the client could not read and a refused redirect carry none, + the redirect because its `Location` names a URL the peer chose. A strictly refused error + envelope keeps them, as it keeps its code. The fields are a copy and are never rendered + into the message. A field sent on several lines reads as one value + joined with ", " in Python and TypeScript; Go keeps the lines, and + `strings.Join(Header.Values(name), ", ")` reads the same value. Go's RPC + `CallError.Status` is now set beside them. TypeScript's `UnaryResponse` gains an optional + `headers`, which the default send fills; an injected send that leaves it out yields + refusals carrying none. The new `refusal-header-vectors.json` pins which failures carry + fields and what they read as, in all three SDKs. +- **Test-only acceptance signers.** A receiver must refuse an acceptance that names an + empty requester even when its signature verifies, and the SDK's own signers refuse to + produce one, so the test proving the refusal needs a signer that does. A test-support + package in each language holds twins of the four acceptance functions that skip only + that refusal: `helperstest.CanonicalAcceptanceBytesAnyRequester`, + `SignOfferAcceptanceAnyRequester`, `CanonicalRequestAcceptanceBytesAnyRequester` and + `SignRequestAcceptanceAnyRequester` (Go, package `sdk/go/helperstest`); + `jcs_acceptance_payload_any_requester`, `sign_offer_acceptance_any_requester`, + `jcs_request_acceptance_payload_any_requester` and + `sign_request_acceptance_any_requester` (Python, `fora_sdk.testing`, not re-exported + from `fora_sdk`); `acceptancePayloadAnyRequester`, `signOfferAcceptanceAnyRequester`, + `requestAcceptancePayloadAnyRequester` and `signRequestAcceptanceAnyRequester` + (TypeScript, export path `./testing`). Every other refusal is kept, in the same text + and order, and the protocol's signers and verifiers are unchanged and still refuse. + Each language renders both through one canonical renderer that takes the requester + check as a parameter: `sdk/go/internal/canonicalsign` in Go, private functions in + Python's `fora_sdk.core`, and `src/acceptance-render.ts` in TypeScript, which neither + package manifest exports. The twins reproduce the `refused` rows of + `acceptance-vectors.json` and `request-acceptance-vectors.json` byte for byte in all + three SDKs, and a test in each fails if production code imports the test package. +- **Strict requests (Python).** `ClientConfig(strict_requests=True)` refuses a request the + contract does not describe, before it is signed. The generated models ignore a member + they do not declare, so a misspelled key in a plain dict passes the default check, is + signed and sent, and is discarded by the server's JSON codec. Under the switch a verb + takes only its own generated request model or a `RawBody`: a plain dict, or a model of + another message, is refused as malformed before anything is resolved, read or sent, + naming the model to pass. The message as stamped, with `ver`, the configured + `requester` and a purchase's reflected offers in it, must then pass the published + strict JSON Schema of the request message and the cross-field rules, so an unknown + member at any depth is refused and named by its path. A `RawBody` is still sent exactly + as given. The switch applies whatever `validation` says and is off by default. A + keyword argument a generated model does not declare is dropped by the model before the + SDK sees it; a type checker reports it. Go verbs take the generated messages and + TypeScript verbs are typed with the request input types, so neither has the switch. + +**Go: a call an interceptor retries reports its last attempt (Go SDK fix; no wire +change).** An interceptor passed to `connect.WithInterceptors` may call `next` again to +retry a call. In v1.1.0 the client then kept what an earlier attempt had received: a call +whose last attempt got no answer reported its Connect code as the peer's +(`CallError.Code`), and under `WithStrictDecoding` an earlier attempt's error envelope could +be checked, and refused, in place of the last attempt's failure. Each request the call +sends is now recorded on its own, so `Code`, and the new `Status` and `Header`, describe +the last request sent. + ## v1.1.0 ### Breaking against v1.0.8 diff --git a/scripts/gen-parity-matrix.py b/scripts/gen-parity-matrix.py index 585d736e..0168d0f6 100755 --- a/scripts/gen-parity-matrix.py +++ b/scripts/gen-parity-matrix.py @@ -67,7 +67,15 @@ ) # Human-readable package headings, in the order Go layers them (L1 -> L2 -> server). -_PKG_ORDER = ["helpers", "resolvers", "core", "connect", "connectserver", "profiles/music"] +_PKG_ORDER = [ + "helpers", + "resolvers", + "core", + "connect", + "connectserver", + "profiles/music", + "helperstest", +] _PKG_TITLE = { "helpers": "helpers — L1 pure primitives (crypto, canonicalization, money, scopes, license terms)", "resolvers": "resolvers — L2 I/O (key/endpoint resolution, active-key, SSRF-guarded fetch)", @@ -75,6 +83,7 @@ "connect": "connect — Connect client + interceptors", "connectserver": "connectserver — server-verify handler binding", "profiles/music": "profiles/music — the fora-music-v1 extension profile type", + "helperstest": "helperstest — test support: acceptance signers that admit an empty requester", } @@ -133,7 +142,7 @@ def _render(parity_map: dict) -> str: w(" Drift-gated in scripts/ci-local.sh. Narrative/rationale: docs/design-history.md. -->") w("") w( - "Go is the oracle (`sdk/go/{helpers,resolvers,core,connect,connectserver,profiles/music}`); " + "Go is the oracle (`sdk/go/{helpers,resolvers,core,connect,connectserver,profiles/music,helperstest}`); " "Python and TS mirror it. This document is **generated** from the same two " "artifacts CI already enforces against the code, so it cannot drift from the " "real surface — a mismatch fails the API-surface gate or the corpus-completeness " diff --git a/sdk/go/README.md b/sdk/go/README.md index 0778e5be..915f6446 100644 --- a/sdk/go/README.md +++ b/sdk/go/README.md @@ -11,13 +11,17 @@ directly with no `replace` directive. | L2 · I/O | **`sdk/go/resolvers`** | the network-fetching tier: well-known JWKS / WBA directory / `fora.json` endpoint / offer-key resolvers, the uncached registration-requirements reader, + the SSRF-guarded HTTP client. Runs on a maintained `net/http` client behind the SSRF guard; composes L1, never the reverse | | L2 · transport | `sdk/go/core` (transport-neutral: Verifier, {verified,rejected}, `DiscoveryResult` per-URI groups, VerifiedOffer guard, signing RoundTripper, ReplayStore — zero Connect) · `sdk/go/connect` (Connect **client** binding: `NewClient` + `NewBrokerClient` + `NewCatalogClient` + `NewAdminClient`; the agent verbs **`Discover` · `Resolve` · `Execute` · `ExecuteBatch` · `ReportUsage` · `Dispute` · `Fetch`**, the Broker purchase **`BrokerClient.Execute`** (offers from several Exchanges in one call), the account-setup verbs **`Register` · `GetAccountStatus`** and the publisher verbs **`PushResources` · `RemoveResources` · `RefreshCatalog`**, the operator verbs **`SetTenantFeeRate` · `SetReportingPolicy` · `RequestDomainVerification` · `ConfirmDomainVerification`** + client options + the `CallError` taxonomy + `ErrorDetailFrom`) · `sdk/go/connectserver` (Connect **server** binding: `NewExchangeServiceHandler` + `NewBrokerServiceHandler` + `NewCatalogServiceHandler` + server options + `AsConnectError` + `AttachErrorDetail`/`AttachDetail` + the reject answer **`RejectCode` · `IsBodyTooLarge` · `WriteReject`**, the one place the 413/429/401 split and the error-envelope body are decided) | transport-neutral core + Connect client/server bindings (state injected) | | L3 | separate packages | framework adapters (convert, never replace) — later | +| test support | `sdk/go/helperstest` | acceptance signers that admit an empty requester, for a test proving a verifier refuses one. Never imported by production code | The `L2` tier is split by kind: the **I/O** package (`resolvers`) is the only tier that dials the network (it holds the maintained HTTP client and the SSRF guard), while the **transport** packages (`core` / `connect` / `connectserver`) hold no dialing surface of their own — `connect` composes the I/O tier's guarded transport and content fetcher rather than opening a second one. An io-leaf guard (`helpers/io_leaf_guard_test.go`) fails the build if any pure -L1 file drags in a dialing surface, so the fetch surface cannot leak back down. +L1 file drags in a dialing surface, so the fetch surface cannot leak back down. It scans +helpers and every package of the module helpers is built from (today +`internal/canonicalsign` and `internal/hostredact`), not one directory, and the +connect-free guards of helpers and core scan their packages' closures the same way. ## L1 — `helpers` @@ -354,12 +358,23 @@ response or re-implements signing: is checked too: the Connect error envelope must carry only `code`, `message` and `details`, name a known Connect code and carry well-formed details, and every `ErrorDetail`, binary `value` and `debug` projection alike, must pass the same two - checks. A refused envelope is `CallMalformed` with `CallError.Code` still set to the - peer's code. + checks. A refused envelope is `CallMalformed` with `CallError.Code`, `Status` and + `Header` still holding the peer's code, HTTP status and header fields. - **`CallError.Code`** is the Connect code of the peer's answer — an error envelope or a non-JSON error status — and zero when no answer arrived. The typed `ErrorDetail` is decoded from the binary `value`; the shared `connect-error-vectors.json` and `error-detail-wire-vectors.json` corpora hold all three SDKs to the same reading. +- **`CallError.Status` and `CallError.Header`** carry the HTTP status and the header + fields of the peer's error answer, 400 or above, on every RPC leg and on `Fetch`, where + `resolvers.FetchError.Header` holds them first: a verifier's `Accept-Signature`, a + gateway's `Retry-After`, an edge's discovery headers for + `helpers.ParseDiscoveryHint(err.Status, err.Header)`. A refused redirect, a 2xx the client + could not read and a failure where nothing answered carry none, and `Error()` never + renders the fields. Read a field sent on several lines as + `strings.Join(err.Header.Values(name), ", ")`, the value the other two SDKs return; + `refusal-header-vectors.json` holds all three to it. When an interceptor passed to + `WithInterceptors` retries a call, `Status`, `Header` and `Code` describe the last + request it sent. - **Delivery URLs are the edge's to check.** `Execute`, `ExecuteBatch` and `BrokerClient.Execute` return every `retrieval_endpoint` exactly as the Exchange issued it, and `Fetch` dials a URL as given with the agent's proof of possession, so @@ -373,6 +388,13 @@ response or re-implements signing: `core.SigningTransportFor` returns a signing transport that signs as the key. `core.WithSignerSource` signs each request as the identity a callback picks, for a service that signs as many agents. +- **Acceptances that name an empty requester.** The `helpers` signers refuse to produce + one, as fora.proto requires of a signer, so `sdk/go/helperstest` holds their twins: + `CanonicalAcceptanceBytesAnyRequester`, `SignOfferAcceptanceAnyRequester`, + `CanonicalRequestAcceptanceBytesAnyRequester` and `SignRequestAcceptanceAnyRequester`. + Each skips only the empty-requester refusal, so the acceptance it signs is valid in every + respect but the requester, and a verifier's refusal of it is attributable to that alone. + Test support only: a test fails if production code imports the package. - **Document readers.** `resolvers.ReadManifest`, `ReadWBADirectory`, `ReadRevocationList` and `ReadLicenseDocument` read what a party publishes, through the guarded client unless `ReadOptions.Client` replaces it, and return a `Document` diff --git a/sdk/go/connect/answer.go b/sdk/go/connect/answer.go index 7986be79..73e64363 100644 --- a/sdk/go/connect/answer.go +++ b/sdk/go/connect/answer.go @@ -7,6 +7,8 @@ import ( "sync/atomic" connectrpc "connectrpc.com/connect" + + "github.com/FORA-Protocol/protocol/sdk/go/internal/failure" ) // CallError.Code is the Connect code of the peer's ANSWER. connect-go hands back a @@ -15,28 +17,76 @@ import ( // not marked as one the server sent, and neither is CodeUnavailable for a dial that // never connected. Python and TypeScript both report a code for the first and none // for the second, so this client records the fact itself: the interceptor below -// puts a holder on the call's context, and the client's transport writes the status -// of whatever response arrived into it. connect-go builds the HTTP request with the -// context the interceptor chain passed down, so the two meet without any verb -// knowing. +// puts a holder on the call's context, and the client's transport writes the +// answer that arrived into it. connect-go builds the HTTP request with the context +// the interceptor chain passed down, so the two meet without any verb knowing. +// +// The same record supplies CallError.Status and CallError.Header. connect-go's own +// Error.Meta() cannot: it is filled only when the body decoded as a Connect error +// envelope, so a gateway's 503 and its Retry-After, which arrive as plain text, +// would carry nothing. +// +// The record belongs to one round trip. An interceptor passed to WithInterceptors runs +// inside answerInterceptor and may call next more than once to retry the call, and +// every request it sends clears the record before it goes out. A failure therefore +// describes the last request the call sent: a transport failure after an earlier error +// answer carries no code, status or header fields. The record cannot tell which attempt +// an error came from, so an interceptor that builds its own error, returns an earlier +// attempt's or sends attempts concurrently gets the last request's record beside it. type answerKey struct{} -// answerHolder carries the HTTP status of the response that arrived, if any. -type answerHolder struct{ status atomic.Int64 } +// answer is the response that arrived on a call: its status, and the header fields +// a failure may carry for it (failure.RefusalHeader). +type answer struct { + status int + header http.Header +} + +// answerHolder carries the answer the call's last request received, if any. It is +// written by the transport and read by the interceptor, so it is held atomically rather +// than relying on the two running on one goroutine. +type answerHolder struct{ last atomic.Pointer[answer] } + +// forget drops what an earlier request on the call recorded. A nil holder, the one a +// request no SDK verb sent has, holds nothing to forget. +func (h *answerHolder) forget() { + if h != nil { + h.last.Store(nil) + } +} + +// record keeps resp's status and the header fields a failure may carry for it. +func (h *answerHolder) record(resp *http.Response) { + if h != nil { + h.last.Store(&answer{ + status: resp.StatusCode, + header: failure.RefusalHeader(resp.StatusCode, resp.Header), + }) + } +} // answeredError marks a failure whose call received an HTTP error status: the -// peer answered, so its Connect code is the peer's. It wraps rather than replaces -// the error, so errors.As still reaches the *connect.Error and its details. -type answeredError struct{ err error } +// peer answered, so its Connect code is the peer's, and the status and header +// fields are those of its refusal. It wraps rather than replaces the error, so +// errors.As still reaches the *connect.Error and its details. +type answeredError struct { + err error + status int + header http.Header +} func (e *answeredError) Error() string { return e.err.Error() } func (e *answeredError) Unwrap() error { return e.err } -// peerAnswered reports whether err came from a call the peer answered. -func peerAnswered(err error) bool { +// peerAnswer returns the error answer err came from, when the peer answered the +// call with one. +func peerAnswer(err error) (*answeredError, bool) { var a *answeredError - return errors.As(err, &a) + if errors.As(err, &a) { + return a, true + } + return nil, false } // answerInterceptor is the outermost client interceptor: it opens the holder the @@ -52,8 +102,8 @@ func (answerInterceptor) WrapUnary(next connectrpc.UnaryFunc) connectrpc.UnaryFu // 3xx is a redirect this client refused to follow — a server that did not // answer the call. Neither carries a verdict of the peer's. if err != nil { - if holder.status.Load() >= http.StatusBadRequest { - return resp, &answeredError{err: err} + if last := holder.last.Load(); last != nil && last.status >= http.StatusBadRequest { + return resp, &answeredError{err: err, status: last.status, header: last.header} } } return resp, err @@ -69,17 +119,26 @@ func (answerInterceptor) WrapStreamingHandler(next connectrpc.StreamingHandlerFu } // answerRecorder is the outermost RoundTripper on every signed leg: it records the -// status of the response that came back into the call's holder, and, under strict -// decoding, the error body into the call's envelope capture. +// answer that came back into the call's holder, and, under strict decoding, the +// error body into the call's envelope capture. The header fields are copied here, +// under failure.RefusalHeader, so the record holds nothing of a redirect's and +// keeps no reference to a response the caller will close. +// +// Both are cleared before the request is sent, not when no answer comes back: a 2xx, +// an error body past the read cap and one in a coding this client did not ask for all +// arrive and store nothing, and none of them may leave an earlier attempt's record in +// place. type answerRecorder struct{ next http.RoundTripper } func (r answerRecorder) RoundTrip(req *http.Request) (*http.Response, error) { + holder, _ := req.Context().Value(answerKey{}).(*answerHolder) + capture := envelopeCaptureOf(req) + holder.forget() + capture.forget() resp, err := r.next.RoundTrip(req) if resp != nil { - if holder, ok := req.Context().Value(answerKey{}).(*answerHolder); ok { - holder.status.Store(int64(resp.StatusCode)) - } - captureErrorBody(req, resp) + holder.record(resp) + captureErrorBody(capture, resp) } return resp, err } diff --git a/sdk/go/connect/answer_internal_test.go b/sdk/go/connect/answer_internal_test.go new file mode 100644 index 00000000..30e2d897 --- /dev/null +++ b/sdk/go/connect/answer_internal_test.go @@ -0,0 +1,185 @@ +package connect + +// The answer recorder keeps the record of one round trip. Each request clears what an +// earlier request on the same call recorded before it is sent, so no answer at all, or +// an answer that stores nothing, leaves nothing of an earlier attempt behind; an answer +// that does store something is recorded as before. + +import ( + "bytes" + "context" + "errors" + "io" + "net/http" + "strings" + "testing" +) + +type roundTripFunc func(*http.Request) (*http.Response, error) + +func (f roundTripFunc) RoundTrip(req *http.Request) (*http.Response, error) { return f(req) } + +// answering is a transport that returns resp, or fails with err. +func answering(resp *http.Response, err error) http.RoundTripper { + return roundTripFunc(func(req *http.Request) (*http.Response, error) { + if resp != nil { + resp.Request = req + } + return resp, err + }) +} + +func answerWith(status int, header http.Header, body io.ReadCloser) *http.Response { + return &http.Response{StatusCode: status, Header: header, Body: body} +} + +const earlierEnvelope = `{"code":"resource_exhausted","message":"slow down","bogus":true}` + +// afterAnEarlierAttempt returns a request on a call whose holder and capture already +// hold an earlier attempt's record: a 503 with its Retry-After, and that answer's error +// body. +func afterAnEarlierAttempt(t *testing.T) (*http.Request, *answerHolder, *envelopeCapture) { + t.Helper() + holder := &answerHolder{} + holder.last.Store(&answer{status: http.StatusServiceUnavailable, header: http.Header{"Retry-After": {"86400"}}}) + capture := &envelopeCapture{} + capture.store([]byte(earlierEnvelope)) + ctx := context.WithValue(context.WithValue(context.Background(), answerKey{}, holder), envelopeKey{}, capture) + req, err := http.NewRequestWithContext(ctx, http.MethodPost, + "https://exchange.test/fora.v1.ExchangeService/DiscoverResources", http.NoBody) + if err != nil { + t.Fatal(err) + } + return req, holder, capture +} + +func assertNoCapture(t *testing.T, capture *envelopeCapture) { + t.Helper() + if body, ok := capture.load(); ok { + t.Errorf("the capture keeps an error body this round trip did not receive: %q", body) + } +} + +func readAll(t *testing.T, body io.Reader) string { + t.Helper() + b, err := io.ReadAll(body) + if err != nil { + t.Fatalf("read the body: %v", err) + } + return string(b) +} + +func TestARoundTripThatGetsNoAnswerLeavesNoRecord(t *testing.T) { + t.Parallel() + req, holder, capture := afterAnEarlierAttempt(t) + reset := errors.New("connection reset by peer") + resp, err := answerRecorder{next: answering(nil, reset)}.RoundTrip(req) + if resp != nil || !errors.Is(err, reset) { + t.Fatalf("got %v, %v; want the transport's failure unchanged", resp, err) + } + if last := holder.last.Load(); last != nil { + t.Errorf("the holder keeps an earlier attempt's answer: %+v", *last) + } + assertNoCapture(t, capture) +} + +func TestARoundTripRecordsOnlyWhatItReceived(t *testing.T) { + t.Parallel() + + t.Run("a success", func(t *testing.T) { + t.Parallel() + req, holder, capture := afterAnEarlierAttempt(t) + ok := answerWith(http.StatusOK, http.Header{"Content-Type": {"application/json"}}, + io.NopCloser(strings.NewReader("{}"))) + if _, err := (answerRecorder{next: answering(ok, nil)}).RoundTrip(req); err != nil { + t.Fatal(err) + } + last := holder.last.Load() + if last == nil || last.status != http.StatusOK || last.header != nil { + t.Errorf("record = %+v, want status 200 and no header fields", last) + } + assertNoCapture(t, capture) + }) + + t.Run("an error answer", func(t *testing.T) { + t.Parallel() + req, holder, capture := afterAnEarlierAttempt(t) + const envelope = `{"code":"unauthenticated","message":"the request is not signed"}` + header := http.Header{"Content-Type": {"application/json"}, "Accept-Signature": {"sig1=()"}} + got, err := answerRecorder{next: answering( + answerWith(http.StatusUnauthorized, header, io.NopCloser(strings.NewReader(envelope))), nil)}.RoundTrip(req) + if err != nil { + t.Fatal(err) + } + last := holder.last.Load() + if last == nil || last.status != http.StatusUnauthorized { + t.Fatalf("record = %+v, want status 401", last) + } + if v := last.header.Get("Retry-After"); v != "" { + t.Errorf("the record keeps the earlier attempt's Retry-After %q", v) + } + header.Set("Accept-Signature", "changed after the round trip") + if v := last.header.Get("Accept-Signature"); v != "sig1=()" { + t.Errorf("Accept-Signature = %q; the record must hold a copy of the answer's fields", v) + } + if body, ok := capture.load(); !ok || string(body) != envelope { + t.Errorf("capture = %q (%v), want this answer's envelope", body, ok) + } + if body := readAll(t, got.Body); body != envelope { + t.Errorf("the body reads %q downstream, want it whole", body) + } + }) +} + +// An error answer can arrive and still store no body: one past the read cap, one in a +// coding this client did not ask for, or one with no body. The earlier attempt's body +// must not stand in for it. +func TestAnErrorAnswerThatStoresNoBodyLeavesNoEarlierOne(t *testing.T) { + t.Parallel() + pastTheCap := strings.Repeat("x", DefaultMaxRPCReadBytes+1) + cases := []struct { + name string + header http.Header + body io.ReadCloser + reads string + }{ + {"past the read cap", http.Header{"Content-Type": {"application/json"}}, + io.NopCloser(strings.NewReader(pastTheCap)), pastTheCap}, + {"in a coding this client did not ask for", http.Header{"Content-Type": {"application/json"}, "Content-Encoding": {"br"}}, + io.NopCloser(bytes.NewReader([]byte{0x0b, 0x02})), "\x0b\x02"}, + {"with no body", http.Header{"Content-Type": {"application/json"}}, nil, ""}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + t.Parallel() + req, holder, capture := afterAnEarlierAttempt(t) + got, err := answerRecorder{next: answering(answerWith(http.StatusServiceUnavailable, c.header, c.body), nil)}.RoundTrip(req) + if err != nil { + t.Fatal(err) + } + if last := holder.last.Load(); last == nil || last.status != http.StatusServiceUnavailable { + t.Errorf("record = %+v, want this answer's status", last) + } + assertNoCapture(t, capture) + if got.Body != nil { + if body := readAll(t, got.Body); body != c.reads { + t.Errorf("the body reads %d bytes downstream, want %d", len(body), len(c.reads)) + } + } + }) + } +} + +func TestARequestNoVerbSentPassesThroughUntouched(t *testing.T) { + t.Parallel() + req, err := http.NewRequestWithContext(context.Background(), http.MethodGet, "https://exchange.test/", http.NoBody) + if err != nil { + t.Fatal(err) + } + body := io.NopCloser(strings.NewReader("draining")) + want := answerWith(http.StatusServiceUnavailable, http.Header{"Retry-After": {"1"}}, body) + got, err := answerRecorder{next: answering(want, nil)}.RoundTrip(req) + if err != nil || got != want || got.Body != body { + t.Fatalf("got %v, %v; want the transport's response untouched", got, err) + } +} diff --git a/sdk/go/connect/callerror.go b/sdk/go/connect/callerror.go index 6085f861..fccf2166 100644 --- a/sdk/go/connect/callerror.go +++ b/sdk/go/connect/callerror.go @@ -2,6 +2,7 @@ package connect import ( "errors" + "net/http" connectrpc "connectrpc.com/connect" @@ -71,10 +72,28 @@ func (k CallErrorKind) String() string { return failure.Name(callErrorKindNames, // are the failures the Exchange would have named had the request been sent. // ErrorDetailFrom reads all three, so a caller branches on one vocabulary whichever // side declined. +// +// Status, Header and Code describe the last request the call sent, so a call that an +// interceptor passed to WithInterceptors retries reports its last attempt. type CallError struct { - Kind CallErrorKind - Op string - Status int // HTTP status when the peer answered; 0 otherwise + Kind CallErrorKind + Op string + // Status is the HTTP status of the answer the failure is about: on an RPC leg, + // the peer's error answer (400 or above), and on the content leg, the edge's + // answer when it was not a success. It is 0 when no such answer arrived — a + // local failure, a transport failure, a refused redirect, or a 2xx this client + // could not read. + Status int + // Header holds the header fields of the peer's error answer, 400 or above, on + // either leg: an Accept-Signature naming what a verifier requires, a + // Retry-After, an edge's discovery headers (helpers.ParseDiscoveryHint reads + // them with Status). It is nil when Status is not an error status, so a refused + // redirect's Location never reaches a caller through it. + // + // The values are the peer's, unvalidated, and Error() never renders them. A + // field sent on several lines keeps each line; strings.Join(Header.Values(name), + // ", ") reads the combined value the Python and TypeScript clients return. + Header http.Header Reason string // the peer's own refusal token when it sent one Detail *forav1.ErrorDetail // PeerMessage is the developer message the peer put on its TYPED reason, when @@ -207,8 +226,10 @@ func sendError(op string, err error) error { out.Kind = CallRefused } out.Reason = cerr.Code().String() - if peerAnswered(err) { + if answered, ok := peerAnswer(err); ok { out.Code = cerr.Code() + out.Status = answered.status + out.Header = answered.header } if detail, ok := errorDetailFromConnect(cerr); ok { out.Detail = detail @@ -240,6 +261,12 @@ func localRefusal(op string, err error) *CallError { // it is part of what was refused. out.Code = strict.code } + // The status and header fields are the answer's, not the envelope's, so a + // refused envelope keeps them as the code is kept. + if answered, ok := peerAnswer(err); ok { + out.Status = answered.status + out.Header = answered.header + } return out } return nil diff --git a/sdk/go/connect/connect_error_corpus_test.go b/sdk/go/connect/connect_error_corpus_test.go index b8f7b089..486104b3 100644 --- a/sdk/go/connect/connect_error_corpus_test.go +++ b/sdk/go/connect/connect_error_corpus_test.go @@ -74,6 +74,11 @@ func TestConnectErrorCorpusReplay(t *testing.T) { if got := callErr.Code.String(); got != v.Code { t.Errorf("code = %q, want %q", got, v.Code) } + // The status of the answer rides beside the code, every row, as the Python + // and TypeScript replays assert it. + if callErr.Status != v.HTTPStatus { + t.Errorf("status = %d, want %d", callErr.Status, v.HTTPStatus) + } assertStrictRead(t, v, callErr) if !ok { return @@ -146,6 +151,9 @@ func assertStrictRead(t *testing.T, v connectErrorVector, lenient *foraconnect.C if got := strict.Code.String(); got != v.Code { t.Errorf("strict: code = %q, want %q", got, v.Code) } + if strict.Status != v.HTTPStatus { + t.Errorf("strict: status = %d, want %d", strict.Status, v.HTTPStatus) + } detail, hasDetail := foraconnect.ErrorDetailFrom(strict) if v.StrictMalformed { if strict.Kind != foraconnect.CallMalformed { diff --git a/sdk/go/connect/doc.go b/sdk/go/connect/doc.go index 95b46603..c900535a 100644 --- a/sdk/go/connect/doc.go +++ b/sdk/go/connect/doc.go @@ -14,8 +14,9 @@ // is the delivery edge's job when Fetch presents it. The client carries what a // conformance harness needs to drive a service through it: // a pre-signing hook (WithBeforeSign), raw mode (WithRawBody), strict response -// decoding (WithStrictDecoding) and the Connect code of every peer answer -// (CallError.Code). +// decoding (WithStrictDecoding), the Connect code of every peer answer +// (CallError.Code), and the status and header fields of every refusal +// (CallError.Status, CallError.Header). // // The account verbs sit on NewClient rather than on a fourth constructor because // they are the same party holding the same key: an account is per-Exchange and diff --git a/sdk/go/connect/gen_refusal_header_vectors_test.go b/sdk/go/connect/gen_refusal_header_vectors_test.go new file mode 100644 index 00000000..81db2462 --- /dev/null +++ b/sdk/go/connect/gen_refusal_header_vectors_test.go @@ -0,0 +1,229 @@ +package connect_test + +// Refusal-header golden-vector emitter: which failures carry the header fields of the +// answer that produced them, and what those fields read as. +// +// A peer's error answer says what it wants next in its header fields — the signature a +// verifier requires, when a draining gateway will take the call again, where a +// publisher's terms are — and all three clients carry them on their typed failure. They +// read them through three different HTTP libraries: net/http canonicalizes names and +// keeps each line of a repeated field, undici lowercases names and decodes values as +// UTF-8, httpx keeps names as sent. So what the corpus pins is the reading a caller sees: +// a field looked up by any spelling of its name, a repeated field joined with ", ", and +// the rule for when there are fields at all — an answer of status 400 or above, on both +// legs, and never a redirect, whose Location names a URL the peer chose, nor a 2xx the +// client could not read. +// +// Each row records the row's own fields only. The served answer also carries Date and +// Content-Length, which change with the run and say nothing about the rule. +// +// The failure class is deliberately not recorded: a 2xx that is not a FORA answer is a +// refusal in Go, which connect-go classifies by its content type, and malformed in the +// two JSON clients. That difference predates this corpus and is not what it pins. +// +// Captured by driving the REAL client against each answer. Verification no-op by default; +// (re)writes under FORA_UPDATE_VECTORS=1. TEST INFRASTRUCTURE. + +import ( + "context" + "net/http" + "net/http/httptest" + "os" + "strings" + "testing" + + forav1 "github.com/FORA-Protocol/protocol/gen/go/fora/v1" + foraconnect "github.com/FORA-Protocol/protocol/sdk/go/connect" + "github.com/FORA-Protocol/protocol/sdk/go/helpers" + "github.com/FORA-Protocol/protocol/sdk/go/internal/vectorio" +) + +const refusalHeaderVectorsPath = "testdata/refusal-header-vectors.json" + +// refusalHeaderVector is one answer a peer gave, and what the client's failure carries. +type refusalHeaderVector struct { + Name string `json:"name"` + // Leg is "rpc" for a Connect call or "content" for the delivery fetch. + Leg string `json:"leg"` + // Strict decodes the answer strictly (an RPC leg only): an envelope the contract + // does not accept is refused, and keeps its fields. + Strict bool `json:"strict"` + Status int `json:"status"` + ContentType string `json:"content_type"` + Body string `json:"body"` + // Headers are the answer's own field lines, in order; a repeated name is a field + // sent on several lines. + Headers [][2]string `json:"headers"` + // Carried maps each of the row's field names, lowercased, to the value the failure + // carries for it, a repeated field joined with ", ". Empty when the failure carries + // no fields at all, which a replay checks of the WHOLE set, not only the row's names. + Carried map[string]string `json:"carried"` + // Hint is helpers.ParseDiscoveryHint over the failure's own status and fields: what a + // caller reads from an edge's 403 without touching the raw answer. + Hint refusalHint `json:"hint"` +} + +type refusalHint struct { + ContentRules string `json:"content_rules"` + ContentRulesState string `json:"content_rules_state"` + Exchange string `json:"exchange"` + ExchangeState string `json:"exchange_state"` +} + +// The answers, before the client has read them. +var refusalHeaderCases = []refusalHeaderVector{ + { + Name: "rpc_401_accept_signature", Leg: "rpc", Status: http.StatusUnauthorized, + ContentType: "application/json", Body: unauthenticatedEnvelope, + Headers: [][2]string{{helpers.AcceptSignatureHeader, helpers.AcceptSignature(false)}}, + }, + { + Name: "rpc_401_strict_envelope_refused", Leg: "rpc", Strict: true, Status: http.StatusUnauthorized, + ContentType: "application/json", + Body: `{"code":"unauthenticated","message":"the request is not signed","bogus":true}`, + Headers: [][2]string{{helpers.AcceptSignatureHeader, helpers.AcceptSignature(false)}}, + }, + { + Name: "rpc_503_retry_after", Leg: "rpc", Status: http.StatusServiceUnavailable, + ContentType: "text/plain", Body: "draining", + Headers: [][2]string{{"Retry-After", "86400"}}, + }, + { + Name: "rpc_400_two_field_lines", Leg: "rpc", Status: http.StatusBadRequest, + ContentType: "application/json", Body: `{"code":"invalid_argument","message":"refused"}`, + Headers: [][2]string{{"X-Fora-Probe", "first"}, {"X-Fora-Probe", "second"}}, + }, + { + Name: "rpc_302_location", Leg: "rpc", Status: http.StatusFound, + ContentType: "text/plain", + Headers: [][2]string{ + {"Location", "https://elsewhere.test/next?token=s3cret"}, {"X-Fora-Probe", "redirect"}, + }, + }, + { + Name: "rpc_200_not_json", Leg: "rpc", Status: http.StatusOK, + ContentType: "text/html", Body: "", + Headers: [][2]string{{"X-Fora-Probe", "success"}}, + }, + { + Name: "content_401_pop_accept_signature", Leg: "content", Status: http.StatusUnauthorized, + ContentType: "application/json", Body: `{"error":"unauthorized","reason":"pop_sig_invalid"}`, + Headers: [][2]string{{helpers.AcceptSignatureHeader, helpers.PoPAcceptSignature}}, + }, + { + Name: "content_403_discovery_hint", Leg: "content", Status: http.StatusForbidden, + ContentType: "application/json", Body: `{"error":"forbidden"}`, + Headers: [][2]string{ + {helpers.ContentRulesHeader, "https://publisher.test/.well-known/fora.json"}, + {helpers.ExchangeHeader, "exchange.test"}, + }, + }, + { + Name: "content_302_location", Leg: "content", Status: http.StatusFound, + ContentType: "text/plain", + Headers: [][2]string{ + {"Location", "https://elsewhere.test/doc?token=s3cret"}, {"X-Fora-Probe", "redirect"}, + }, + }, +} + +func TestGenerateRefusalHeaderVectors(t *testing.T) { + doc := map[string]any{ + "note": "Which failures carry the header fields of the answer that produced them, " + + "captured from the real Go client: an answer of status 400 or above on either " + + "leg, and never a redirect or a 2xx the client could not read. Each row records " + + "its own field lines only; carried maps each lowercased name to the value read, " + + "a repeated field joined with \", \", and is empty when no field is carried at all.", + "refusals": buildRefusalHeaderVectors(t), + } + if os.Getenv("FORA_UPDATE_VECTORS") == "1" { + if err := vectorio.Write(refusalHeaderVectorsPath, doc); err != nil { + t.Fatalf("write %s: %v", refusalHeaderVectorsPath, err) + } + return + } + stale, err := vectorio.Stale(refusalHeaderVectorsPath, doc) + if err != nil { + t.Fatalf("read %s: %v", refusalHeaderVectorsPath, err) + } + if stale { + t.Fatalf("%s is stale; re-run with FORA_UPDATE_VECTORS=1 to regenerate", refusalHeaderVectorsPath) + } +} + +func buildRefusalHeaderVectors(t *testing.T) []refusalHeaderVector { + t.Helper() + out := make([]refusalHeaderVector, 0, len(refusalHeaderCases)) + for _, c := range refusalHeaderCases { + cerr := refusalHeaderFailure(t, c) + c.Carried, c.Hint = refusalHeaderOutcome(c, cerr) + // The emitter checks the rule it records, so a corpus that broke it could not be + // written in the first place. + if (len(c.Carried) > 0) != (c.Status >= http.StatusBadRequest) { + t.Fatalf("%s: status %d carries %v", c.Name, c.Status, c.Carried) + } + if len(c.Carried) > 0 && cerr.Status != c.Status { + t.Fatalf("%s: carried fields beside status %d, want %d", c.Name, cerr.Status, c.Status) + } + for name, value := range c.Carried { + if strings.Contains(cerr.Error(), value) { + t.Fatalf("%s: the message renders %s: %s", c.Name, name, cerr.Error()) + } + } + out = append(out, c) + } + return out +} + +// refusalHeaderFailure serves the row's answer and drives the real client at it. +func refusalHeaderFailure(t *testing.T, v refusalHeaderVector) *foraconnect.CallError { + t.Helper() + lines := make([]fieldLine, 0, len(v.Headers)) + for _, h := range v.Headers { + lines = append(lines, fieldLine{h[0], h[1]}) + } + srv := httptest.NewServer(refusalHandler(v.Status, v.ContentType, v.Body, lines...)) + defer srv.Close() + + ctx := context.Background() + switch v.Leg { + case "rpc": + var opts []foraconnect.ClientOption + if v.Strict { + opts = append(opts, foraconnect.WithStrictDecoding()) + } + _, err := foraconnect.NewClient(srv.URL, opts...). + Discover(ctx, &forav1.ResourceQuery{Exchange: "exchange.test"}) + return callErrorFrom(t, err) + case "content": + sig := newSigningFixture(t) + client := foraconnect.NewClient("http://home.invalid", + append(allowLoopback(t), + foraconnect.WithSigner(sig.signer), foraconnect.WithSignatureAgent("https://agent.test"), + foraconnect.WithAgentKey(sig.pub), + )...) + _, err := client.Fetch(ctx, srv.URL+"/doc?agent_id=tp") + return callErrorFrom(t, err) + } + t.Fatalf("%s: unknown leg %q", v.Name, v.Leg) + return nil +} + +// refusalHeaderOutcome reads what the failure carries for the row's own field names, and +// the discovery hint a caller parses from it. +func refusalHeaderOutcome(v refusalHeaderVector, cerr *foraconnect.CallError) (map[string]string, refusalHint) { + carried := map[string]string{} + if len(cerr.Header) > 0 { + for _, h := range v.Headers { + name := strings.ToLower(h[0]) + carried[name] = fieldValue(cerr.Header, name) + } + } + hint := helpers.ParseDiscoveryHint(cerr.Status, cerr.Header) + return carried, refusalHint{ + ContentRules: hint.ContentRules, + ContentRulesState: hint.ContentRulesState.String(), + Exchange: hint.Exchange, + ExchangeState: hint.ExchangeState.String(), + } +} diff --git a/sdk/go/connect/last_attempt_test.go b/sdk/go/connect/last_attempt_test.go new file mode 100644 index 00000000..ac8fbec7 --- /dev/null +++ b/sdk/go/connect/last_attempt_test.go @@ -0,0 +1,158 @@ +package connect_test + +// A call an interceptor retries reports its last attempt. The client records what each +// request it sends receives, and clears that record before every request, so a failure +// never describes an earlier attempt: not its status, not its header fields, not the +// judgment that the peer answered, and, under strict decoding, not its error envelope. +// +// The base transport is scripted rather than served, so each attempt's outcome is exact; +// the client composes its own transports on top of it, so all of them run. + +import ( + "context" + "io" + "net/http" + "strconv" + "strings" + "sync/atomic" + "testing" + + connectrpc "connectrpc.com/connect" + + forav1 "github.com/FORA-Protocol/protocol/gen/go/fora/v1" + foraconnect "github.com/FORA-Protocol/protocol/sdk/go/connect" + "github.com/FORA-Protocol/protocol/sdk/go/helpers" +) + +// attempt is what the scripted transport does with one request: answer it, or fail it +// the way a dropped connection does. +type attempt func(*http.Request) (*http.Response, error) + +// answers serves one answer: the status, the content type, the body and each header +// line in order. +func answers(status int, contentType, body string, lines ...fieldLine) attempt { + return func(req *http.Request) (*http.Response, error) { + header := http.Header{"Content-Type": {contentType}} + for _, line := range lines { + header.Add(line.name, line.value) + } + return &http.Response{ + Status: strconv.Itoa(status) + " " + http.StatusText(status), + StatusCode: status, + Proto: "HTTP/1.1", + ProtoMajor: 1, + ProtoMinor: 1, + Header: header, + Body: io.NopCloser(strings.NewReader(body)), + ContentLength: int64(len(body)), + Request: req, + }, nil + } +} + +// drops fails the request with no answer at all, as a connection reset does. +func drops(*http.Request) (*http.Response, error) { return nil, io.ErrUnexpectedEOF } + +// scripted answers the nth request with the nth attempt, and repeats the last attempt +// for any request past the script. +type scripted struct { + attempts []attempt + sent atomic.Int32 +} + +func (s *scripted) RoundTrip(req *http.Request) (*http.Response, error) { + n := int(s.sent.Add(1)) - 1 + if req.Body != nil { + _, _ = io.Copy(io.Discard, req.Body) + _ = req.Body.Close() + } + return s.attempts[min(n, len(s.attempts)-1)](req) +} + +// retryOnce is an application interceptor that sends a failed call once more. +var retryOnce = connectrpc.UnaryInterceptorFunc(func(next connectrpc.UnaryFunc) connectrpc.UnaryFunc { + return func(ctx context.Context, req connectrpc.AnyRequest) (connectrpc.AnyResponse, error) { + if resp, err := next(ctx, req); err == nil { + return resp, nil + } + return next(ctx, req) + } +}) + +// retriedFailure runs one Discover through retryOnce against the two attempts and +// returns its typed failure. +func retriedFailure(t *testing.T, first, second attempt, opts ...foraconnect.ClientOption) *foraconnect.CallError { + t.Helper() + script := &scripted{attempts: []attempt{first, second}} + opts = append(opts, + foraconnect.WithHTTPClient(&http.Client{Transport: script}), + foraconnect.WithInterceptors(retryOnce)) + _, err := foraconnect.NewClient("https://exchange.test", opts...). + Discover(context.Background(), &forav1.ResourceQuery{Exchange: "exchange.test"}) + if sent := script.sent.Load(); sent != 2 { + t.Fatalf("the call sent %d requests, want 2", sent) + } + return callErrorFrom(t, err) +} + +var retryAfterLine = fieldLine{"Retry-After", "86400"} + +func TestARetriedCallReportsItsLastAttempt(t *testing.T) { + t.Parallel() + + t.Run("an answer and then none carries nothing of the answer", func(t *testing.T) { + t.Parallel() + cerr := retriedFailure(t, + answers(http.StatusServiceUnavailable, "text/plain", "draining", retryAfterLine), + drops) + if cerr.Kind != foraconnect.CallUnreachable { + t.Errorf("Kind = %v, want CallUnreachable: %v", cerr.Kind, cerr) + } + if cerr.Status != 0 { + t.Errorf("Status = %d, want 0: the last request got no answer", cerr.Status) + } + if cerr.Header != nil { + t.Errorf("Header = %v, want nil", cerr.Header) + } + if cerr.Code != 0 { + t.Errorf("Code = %v, want none: no peer answered the last request", cerr.Code) + } + if msg := cerr.Error(); strings.Contains(msg, "HTTP 503") { + t.Errorf("the message names the earlier attempt's status: %s", msg) + } + }) + + t.Run("two answers report the second, and only its fields", func(t *testing.T) { + t.Parallel() + cerr := retriedFailure(t, + answers(http.StatusServiceUnavailable, "text/plain", "draining", retryAfterLine), + answers(http.StatusUnauthorized, "application/json", unauthenticatedEnvelope, acceptSignatureLine)) + if cerr.Kind != foraconnect.CallRefused || cerr.Status != http.StatusUnauthorized { + t.Fatalf("got %v with status %d, want a 401 refusal: %v", cerr.Kind, cerr.Status, cerr) + } + if cerr.Code != connectrpc.CodeUnauthenticated { + t.Errorf("Code = %v, want unauthenticated", cerr.Code) + } + if got := fieldValue(cerr.Header, helpers.AcceptSignatureHeader); got != helpers.AcceptSignature(false) { + t.Errorf("Accept-Signature = %q, want %q", got, helpers.AcceptSignature(false)) + } + if got := cerr.Header.Values("Retry-After"); got != nil { + t.Errorf("Retry-After = %q, want none: it was the earlier attempt's", got) + } + }) + + t.Run("under strict decoding an earlier envelope is not judged in place of the last request", func(t *testing.T) { + t.Parallel() + cerr := retriedFailure(t, + answers(http.StatusTooManyRequests, "application/json", + `{"code":"resource_exhausted","message":"slow down","bogus":true}`, retryAfterLine), + drops, + foraconnect.WithStrictDecoding()) + if cerr.Kind != foraconnect.CallUnreachable { + t.Errorf("Kind = %v, want CallUnreachable: the last request got no answer to refuse: %v", cerr.Kind, cerr) + } + if cerr.Status != 0 || cerr.Code != 0 { + t.Errorf("Status = %d, Code = %v; want neither", cerr.Status, cerr.Code) + } + }) +} diff --git a/sdk/go/connect/options.go b/sdk/go/connect/options.go index 3f1c85ad..6d7c9053 100644 --- a/sdk/go/connect/options.go +++ b/sdk/go/connect/options.go @@ -106,6 +106,11 @@ func WithRequestIDFunc(fn core.RequestIDFunc) ClientOption { // WithInterceptors appends application interceptors (tracing, metrics) to the client // SDK stack. They run inside the SDK cross-cutting interceptors. +// +// An interceptor that calls next more than once retries the call. A failed call's +// CallError then describes the last request the call sent, Status, Header and Code +// included, so a retrying interceptor returns that attempt's error and sends its +// attempts one at a time. func WithInterceptors(is ...connectrpc.Interceptor) ClientOption { return func(c *clientConfig) { c.extra = append(c.extra, is...) } } diff --git a/sdk/go/connect/redirect_redaction_test.go b/sdk/go/connect/redirect_redaction_test.go new file mode 100644 index 00000000..2892b8eb --- /dev/null +++ b/sdk/go/connect/redirect_redaction_test.go @@ -0,0 +1,101 @@ +package connect_test + +// A refused redirect never names its target in an RPC error. Every RPC leg refuses +// to follow a 3xx, and the refusal's own message redacts the target to its scheme, +// host and path. net/http wraps that refusal in a *url.Error carrying the raw +// Location the peer sent, credentials included; connect-go discards that wrapper +// before the error reaches this client, which is what keeps the Location out today. +// This pins the result on every face, so a dependency that stopped discarding it +// fails here rather than in a log. The content leg rebuilds the wrapper itself and +// is pinned in the resolvers suite. + +import ( + "context" + "net/http" + "strings" + "testing" + + foraadminv1 "github.com/FORA-Protocol/protocol/gen/go/fora/admin/v1" + forav1 "github.com/FORA-Protocol/protocol/gen/go/fora/v1" + foraconnect "github.com/FORA-Protocol/protocol/sdk/go/connect" +) + +// redirectTarget carries a credential in every part a URL can carry one: userinfo, +// the query and the fragment. +const redirectTarget = "https://user:pw-6c1f0e@elsewhere.test/next?token=s3cret-6c1f0e#frag-6c1f0e" + +var redirectSecrets = []string{"pw-6c1f0e", "s3cret-6c1f0e", "frag-6c1f0e"} + +// assertRedirectRedacted fails unless err is the refusal of a redirect whose message +// names no secret of the target, while still naming where it pointed. +func assertRedirectRedacted(t *testing.T, err error) { + t.Helper() + cerr := callErrorFrom(t, err) + if cerr.Kind != foraconnect.CallUnreachable { + t.Errorf("Kind = %v, want CallUnreachable: %v", cerr.Kind, cerr) + } + msg := cerr.Error() + for _, secret := range redirectSecrets { + if strings.Contains(msg, secret) { + t.Errorf("the message names %q from the redirect target: %s", secret, msg) + } + } + if !strings.Contains(msg, "elsewhere.test/next") { + t.Errorf("the message no longer says where the redirect pointed: %s", msg) + } +} + +func TestARefusedRedirectNeverNamesItsTarget(t *testing.T) { + t.Parallel() + ctx := context.Background() + faces := []struct { + name string + call func(baseURL string) error + }{ + {"discover", func(baseURL string) error { + _, err := foraconnect.NewClient(baseURL).Discover(ctx, &forav1.ResourceQuery{Exchange: "exchange.test"}) + return err + }}, + {"a raw body", func(baseURL string) error { + _, err := foraconnect.NewClient(baseURL).Discover(ctx, nil, foraconnect.WithRawBody([]byte{})) + return err + }}, + {"the Broker", func(baseURL string) error { + _, err := foraconnect.NewBrokerClient(baseURL, foraconnect.WithRequester(testRequester())). + Resolve(ctx, &forav1.DiscoveryRequest{}) + return err + }}, + {"the catalog", func(baseURL string) error { + _, err := foraconnect.NewCatalogClient(baseURL). + PushResources(ctx, &forav1.PushResourcesRequest{Exchange: "exchange.test"}) + return err + }}, + {"the operator plane", func(baseURL string) error { + _, err := foraconnect.NewAdminClient(baseURL).SetTenantFeeRate(ctx, &foraadminv1.SetTenantFeeRateRequest{}) + return err + }}, + } + for _, face := range faces { + t.Run(face.name, func(t *testing.T) { + t.Parallel() + srv := refusalAnswer(t, http.StatusFound, "text/plain", "", fieldLine{"Location", redirectTarget}) + assertRedirectRedacted(t, face.call(srv.URL)) + }) + } +} + +// TestTheOfferDerivedLegNeverNamesARedirectsTarget covers the leg whose host +// another party chose. Not parallel: the loopback posture is set in the environment. +func TestTheOfferDerivedLegNeverNamesARedirectsTarget(t *testing.T) { + sig := newSigningFixture(t) + domain, _ := loopbackManifestServer(t, + refusalHandler(http.StatusFound, "text/plain", "", fieldLine{"Location", redirectTarget})) + client := foraconnect.NewClient("http://home.invalid", + append(allowLoopback(t), foraconnect.WithSigner(sig.signer), foraconnect.WithSignatureAgent("https://agent.test"))...) + _, err := client.ReportUsage(context.Background(), &forav1.UsageReport{ + Exchange: domain, + TransactionId: "txn-1", + Usage: &forav1.Usage{Function: []string{"ai-input"}}, + }) + assertRedirectRedacted(t, err) +} diff --git a/sdk/go/connect/refusal_header_corpus_test.go b/sdk/go/connect/refusal_header_corpus_test.go new file mode 100644 index 00000000..cd506499 --- /dev/null +++ b/sdk/go/connect/refusal_header_corpus_test.go @@ -0,0 +1,68 @@ +package connect_test + +// Go replay of the refusal-header corpus (refusal-header-vectors.json). +// +// The corpus is the cross-language contract for which failures carry the header fields of +// the answer that produced them, so Go replays the committed rows too rather than only +// emitting them: each row's own answer is served again and the real client must carry +// exactly what the row records. + +import ( + "encoding/json" + "net/http" + "os" + "testing" +) + +func TestRefusalHeaderCorpusReplay(t *testing.T) { + raw, err := os.ReadFile(refusalHeaderVectorsPath) //nolint:gosec // a committed test vector + if err != nil { + t.Fatalf("read %s: %v", refusalHeaderVectorsPath, err) + } + var doc struct { + Refusals []refusalHeaderVector `json:"refusals"` + } + if err := json.Unmarshal(raw, &doc); err != nil { + t.Fatalf("parse %s: %v", refusalHeaderVectorsPath, err) + } + if len(doc.Refusals) == 0 { + t.Fatalf("%s carries no vectors — the replay would assert nothing", refusalHeaderVectorsPath) + } + + // Both outcomes on both legs, or the corpus pins half a rule. + seen := map[string]bool{} + for _, row := range doc.Refusals { + t.Run(row.Name, func(t *testing.T) { + cerr := refusalHeaderFailure(t, row) + if len(row.Carried) == 0 { + if len(cerr.Header) != 0 { + t.Errorf("carries %v, want no field at all", cerr.Header) + } + } else { + if cerr.Status != row.Status { + t.Errorf("status = %d, want %d", cerr.Status, row.Status) + } + for name, want := range row.Carried { + if got := fieldValue(cerr.Header, name); got != want { + t.Errorf("%s = %q, want %q", name, got, want) + } + } + } + if _, hint := refusalHeaderOutcome(row, cerr); hint != row.Hint { + t.Errorf("hint = %+v, want %+v", hint, row.Hint) + } + }) + seen[row.Leg+map[bool]string{true: " carried", false: " empty"}[len(row.Carried) > 0]] = true + if row.Strict { + seen["strict"] = true + } + if row.Status == http.StatusForbidden && row.Hint.ExchangeState == "valid" { + seen["hint"] = true + } + } + for _, want := range []string{"rpc carried", "rpc empty", "content carried", "content empty", "strict", "hint"} { + if !seen[want] { + t.Errorf("the corpus has no %s row", want) + } + } +} diff --git a/sdk/go/connect/refusal_header_test.go b/sdk/go/connect/refusal_header_test.go new file mode 100644 index 00000000..d434c579 --- /dev/null +++ b/sdk/go/connect/refusal_header_test.go @@ -0,0 +1,368 @@ +package connect_test + +// A refusal carries the status and header fields of the answer that refused, and +// only a refusal does. An error answer, 400 or above, puts both on the CallError on +// every RPC face and on the delivery fetch; a redirect, a 2xx the client could not +// read, and a failure where nothing answered carry neither. The redirect is the case +// the rule exists for: its Location names a URL the peer chose, so a refused redirect +// must not hand its target back through the header fields. +// +// Every case runs the real client against a real HTTP answer. + +import ( + "context" + "errors" + "net/http" + "net/http/httptest" + "strings" + "testing" + + connectrpc "connectrpc.com/connect" + + foraadminv1 "github.com/FORA-Protocol/protocol/gen/go/fora/admin/v1" + forav1 "github.com/FORA-Protocol/protocol/gen/go/fora/v1" + foraconnect "github.com/FORA-Protocol/protocol/sdk/go/connect" + foraserver "github.com/FORA-Protocol/protocol/sdk/go/connectserver" + "github.com/FORA-Protocol/protocol/sdk/go/helpers" +) + +// fieldLine is one header line an answer is served with. +type fieldLine struct{ name, value string } + +// refusalAnswer serves one fixed answer to every request: the status, the content +// type, the body and each header line in order, a repeated name as repeated lines. +func refusalAnswer(t *testing.T, status int, contentType, body string, lines ...fieldLine) *httptest.Server { + t.Helper() + srv := httptest.NewServer(refusalHandler(status, contentType, body, lines...)) + t.Cleanup(srv.Close) + return srv +} + +func refusalHandler(status int, contentType, body string, lines ...fieldLine) http.Handler { + return http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set("Content-Type", contentType) + for _, line := range lines { + w.Header().Add(line.name, line.value) + } + w.WriteHeader(status) + _, _ = w.Write([]byte(body)) + }) +} + +func callErrorFrom(t *testing.T, err error) *foraconnect.CallError { + t.Helper() + var cerr *foraconnect.CallError + if !errors.As(err, &cerr) { + t.Fatalf("not a typed failure: %v", err) + } + return cerr +} + +// fieldValue reads a field the way the Python and TypeScript clients return it: every +// line of it, joined with ", ". +func fieldValue(h http.Header, name string) string { + return strings.Join(h.Values(name), ", ") +} + +// assertFieldsNotRendered fails when a carried value appears in the error's message. +// The values are the peer's, so a log line must never repeat them. +func assertFieldsNotRendered(t *testing.T, cerr *foraconnect.CallError) { + t.Helper() + for name, values := range cerr.Header { + for _, value := range values { + if strings.Contains(cerr.Error(), value) { + t.Errorf("the message renders the %s value %q: %s", name, value, cerr.Error()) + } + } + } +} + +const ( + unauthenticatedEnvelope = `{"code":"unauthenticated","message":"the request is not signed"}` + // headerProbe is a value no message carries by accident, so its absence from + // Error() is evidence rather than luck. + headerProbe = "probe-6c1f0e" +) + +var acceptSignatureLine = fieldLine{helpers.AcceptSignatureHeader, helpers.AcceptSignature(false)} + +func TestARefusalCarriesItsStatusAndHeaderFields(t *testing.T) { + t.Parallel() + cases := []struct { + name string + status int + contentType string + body string + lines []fieldLine + kind foraconnect.CallErrorKind + want map[string]string + }{ + { + name: "an envelope refusal names what the verifier requires", + status: http.StatusUnauthorized, contentType: "application/json", body: unauthenticatedEnvelope, + lines: []fieldLine{acceptSignatureLine}, + kind: foraconnect.CallRefused, + want: map[string]string{helpers.AcceptSignatureHeader: helpers.AcceptSignature(false)}, + }, + { + name: "a draining gateway says when to come back", + status: http.StatusServiceUnavailable, contentType: "text/plain", body: "draining", + lines: []fieldLine{{"Retry-After", "86400"}}, + kind: foraconnect.CallUnreachable, + want: map[string]string{"Retry-After": "86400"}, + }, + { + name: "a field sent on two lines keeps both", + status: http.StatusBadRequest, contentType: "application/json", + body: `{"code":"invalid_argument","message":"refused"}`, + lines: []fieldLine{{"X-Fora-Probe", headerProbe + "-first"}, {"X-Fora-Probe", headerProbe + "-second"}}, + kind: foraconnect.CallRefused, + want: map[string]string{"X-Fora-Probe": headerProbe + "-first, " + headerProbe + "-second"}, + }, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + t.Parallel() + srv := refusalAnswer(t, c.status, c.contentType, c.body, c.lines...) + _, err := foraconnect.NewClient(srv.URL).Discover(context.Background(), &forav1.ResourceQuery{Exchange: "exchange.test"}) + cerr := callErrorFrom(t, err) + if cerr.Kind != c.kind { + t.Errorf("Kind = %v, want %v", cerr.Kind, c.kind) + } + if cerr.Status != c.status { + t.Errorf("Status = %d, want %d", cerr.Status, c.status) + } + for name, want := range c.want { + if got := fieldValue(cerr.Header, name); got != want { + t.Errorf("%s = %q, want %q", name, got, want) + } + } + // Lowercase reads the same field: http.Header canonicalizes on read. + for name, want := range c.want { + if got := fieldValue(cerr.Header, strings.ToLower(name)); got != want { + t.Errorf("lowercase %s = %q, want %q", name, got, want) + } + } + assertFieldsNotRendered(t, cerr) + }) + } +} + +// TestARefusalsStatusIsRendered pins the visible half of the status: an RPC refusal +// now names it, as the content leg's always did. +func TestARefusalsStatusIsRendered(t *testing.T) { + t.Parallel() + srv := refusalAnswer(t, http.StatusUnauthorized, "application/json", unauthenticatedEnvelope, acceptSignatureLine) + _, err := foraconnect.NewClient(srv.URL).Discover(context.Background(), &forav1.ResourceQuery{Exchange: "exchange.test"}) + if msg := callErrorFrom(t, err).Error(); !strings.Contains(msg, "(HTTP 401 Unauthorized)") { + t.Errorf("the message does not name the status: %s", msg) + } +} + +func TestARedirectOrAnUnreadableSuccessCarriesNoStatusOrHeaderFields(t *testing.T) { + t.Parallel() + cases := []struct { + name string + status int + contentType string + body string + lines []fieldLine + }{ + { + name: "a refused redirect", + status: http.StatusFound, contentType: "text/plain", + lines: []fieldLine{ + {"Location", "https://elsewhere.test/next?token=" + headerProbe}, + {"X-Fora-Probe", headerProbe}, + }, + }, + { + name: "a 2xx that is not a FORA answer", + status: http.StatusOK, contentType: "text/html", body: "", + lines: []fieldLine{{"X-Fora-Probe", headerProbe}}, + }, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + t.Parallel() + srv := refusalAnswer(t, c.status, c.contentType, c.body, c.lines...) + _, err := foraconnect.NewClient(srv.URL).Discover(context.Background(), &forav1.ResourceQuery{Exchange: "exchange.test"}) + cerr := callErrorFrom(t, err) + if cerr.Status != 0 { + t.Errorf("Status = %d, want 0", cerr.Status) + } + if cerr.Header != nil { + t.Errorf("Header = %v, want nil", cerr.Header) + } + }) + } +} + +func TestAStrictlyRefusedEnvelopeKeepsItsStatusAndHeaderFields(t *testing.T) { + t.Parallel() + // An unknown envelope member: the answer is refused, and what the peer said about + // itself — its code, its status, its header fields — stays readable. + srv := refusalAnswer(t, http.StatusUnauthorized, "application/json", + `{"code":"unauthenticated","message":"the request is not signed","bogus":true}`, acceptSignatureLine) + _, err := foraconnect.NewClient(srv.URL, foraconnect.WithStrictDecoding()). + Discover(context.Background(), &forav1.ResourceQuery{Exchange: "exchange.test"}) + cerr := callErrorFrom(t, err) + if cerr.Kind != foraconnect.CallMalformed { + t.Fatalf("Kind = %v, want CallMalformed: %v", cerr.Kind, err) + } + if cerr.Code != connectrpc.CodeUnauthenticated { + t.Errorf("Code = %v, want unauthenticated", cerr.Code) + } + if cerr.Status != http.StatusUnauthorized { + t.Errorf("Status = %d, want 401", cerr.Status) + } + if got := fieldValue(cerr.Header, helpers.AcceptSignatureHeader); got != helpers.AcceptSignature(false) { + t.Errorf("Accept-Signature = %q, want %q", got, helpers.AcceptSignature(false)) + } +} + +func TestEveryRPCFaceCarriesARefusalsHeaderFields(t *testing.T) { + t.Parallel() + ctx := context.Background() + faces := []struct { + name string + call func(baseURL string) error + }{ + {"discover", func(baseURL string) error { + _, err := foraconnect.NewClient(baseURL).Discover(ctx, &forav1.ResourceQuery{Exchange: "exchange.test"}) + return err + }}, + {"a raw body", func(baseURL string) error { + _, err := foraconnect.NewClient(baseURL).Discover(ctx, nil, foraconnect.WithRawBody([]byte{})) + return err + }}, + {"the Broker", func(baseURL string) error { + _, err := foraconnect.NewBrokerClient(baseURL, foraconnect.WithRequester(testRequester())). + Resolve(ctx, &forav1.DiscoveryRequest{}) + return err + }}, + {"the catalog", func(baseURL string) error { + _, err := foraconnect.NewCatalogClient(baseURL). + PushResources(ctx, &forav1.PushResourcesRequest{Exchange: "exchange.test"}) + return err + }}, + {"the operator plane", func(baseURL string) error { + _, err := foraconnect.NewAdminClient(baseURL).SetTenantFeeRate(ctx, &foraadminv1.SetTenantFeeRateRequest{}) + return err + }}, + } + for _, face := range faces { + t.Run(face.name, func(t *testing.T) { + t.Parallel() + srv := refusalAnswer(t, http.StatusUnauthorized, "application/json", unauthenticatedEnvelope, acceptSignatureLine) + cerr := callErrorFrom(t, face.call(srv.URL)) + if cerr.Status != http.StatusUnauthorized { + t.Errorf("Status = %d, want 401: %v", cerr.Status, cerr) + } + if got := fieldValue(cerr.Header, helpers.AcceptSignatureHeader); got != helpers.AcceptSignature(false) { + t.Errorf("Accept-Signature = %q, want %q", got, helpers.AcceptSignature(false)) + } + }) + } +} + +// TestTheOfferDerivedLegCarriesARefusalsHeaderFields covers the leg a usage report +// takes: an Exchange the offer named, resolved from its own manifest, over the +// guarded transport. Not parallel: the loopback posture is set in the environment. +func TestTheOfferDerivedLegCarriesARefusalsHeaderFields(t *testing.T) { + sig := newSigningFixture(t) + domain, _ := loopbackManifestServer(t, + refusalHandler(http.StatusUnauthorized, "application/json", unauthenticatedEnvelope, acceptSignatureLine)) + client := foraconnect.NewClient("http://home.invalid", + append(allowLoopback(t), foraconnect.WithSigner(sig.signer), foraconnect.WithSignatureAgent("https://agent.test"))...) + _, err := client.ReportUsage(context.Background(), &forav1.UsageReport{ + Exchange: domain, + TransactionId: "txn-1", + Usage: &forav1.Usage{Function: []string{"ai-input"}}, + }) + cerr := callErrorFrom(t, err) + if cerr.Status != http.StatusUnauthorized { + t.Errorf("Status = %d, want 401: %v", cerr.Status, cerr) + } + if got := fieldValue(cerr.Header, helpers.AcceptSignatureHeader); got != helpers.AcceptSignature(false) { + t.Errorf("Accept-Signature = %q, want %q", got, helpers.AcceptSignature(false)) + } +} + +// TestTheSDKsOwnServerRefusalIsReadable closes the loop with the server binding: an +// unsigned call to a real verifying handler is answered 401, and the Accept-Signature +// that handler writes reaches the caller on the CallError. +func TestTheSDKsOwnServerRefusalIsReadable(t *testing.T) { + t.Parallel() + sig := newSigningFixture(t) + path, h := foraserver.NewExchangeServiceHandler(&stubExchange{}, foraserver.WithKeyResolver(sig.resolver)) + mux := http.NewServeMux() + mux.Handle(path, h) + srv := httptest.NewServer(mux) + t.Cleanup(srv.Close) + + _, err := foraconnect.NewClient(srv.URL).Discover(context.Background(), &forav1.ResourceQuery{Exchange: "exchange.test"}) + cerr := callErrorFrom(t, err) + if cerr.Kind != foraconnect.CallRefused || cerr.Status != http.StatusUnauthorized { + t.Fatalf("got %v with status %d, want a 401 refusal: %v", cerr.Kind, cerr.Status, cerr) + } + if got := cerr.Header.Get(helpers.AcceptSignatureHeader); got != helpers.AcceptSignature(false) { + t.Errorf("Accept-Signature = %q, want %q", got, helpers.AcceptSignature(false)) + } +} + +// TestTheDeliveryFetchCarriesTheEdgesRefusalFields covers the content leg: a refused +// proof is answered with the Accept-Signature that asks for the right one, a 403 +// carries the discovery headers helpers.ParseDiscoveryHint reads, and a redirect +// carries nothing. Not parallel: the loopback posture is set in the environment. +func TestTheDeliveryFetchCarriesTheEdgesRefusalFields(t *testing.T) { + sig := newSigningFixture(t) + client := foraconnect.NewClient("http://home.invalid", + append(allowLoopback(t), + foraconnect.WithSigner(sig.signer), foraconnect.WithSignatureAgent("https://agent.test"), + foraconnect.WithAgentKey(sig.pub), + )...) + fetch := func(t *testing.T, srv *httptest.Server) *foraconnect.CallError { + t.Helper() + _, err := client.Fetch(context.Background(), srv.URL+"/doc?agent_id=tp") + return callErrorFrom(t, err) + } + + t.Run("a refused proof is asked for again", func(t *testing.T) { + edge := refusalAnswer(t, http.StatusUnauthorized, "application/json", + `{"error":"unauthorized","reason":"pop_sig_invalid"}`, + fieldLine{helpers.AcceptSignatureHeader, helpers.PoPAcceptSignature}) + cerr := fetch(t, edge) + if cerr.Status != http.StatusUnauthorized { + t.Errorf("Status = %d, want 401", cerr.Status) + } + if got := fieldValue(cerr.Header, helpers.AcceptSignatureHeader); got != helpers.PoPAcceptSignature { + t.Errorf("Accept-Signature = %q, want %q", got, helpers.PoPAcceptSignature) + } + assertFieldsNotRendered(t, cerr) + }) + + t.Run("a 403 names where the terms are", func(t *testing.T) { + const rules = "https://publisher.test/.well-known/fora.json" + edge := refusalAnswer(t, http.StatusForbidden, "application/json", `{"error":"forbidden"}`, + fieldLine{helpers.ContentRulesHeader, rules}, fieldLine{helpers.ExchangeHeader, "exchange.test"}) + cerr := fetch(t, edge) + hint := helpers.ParseDiscoveryHint(cerr.Status, cerr.Header) + if hint.ContentRulesState != helpers.HintValid || hint.ContentRules != rules { + t.Errorf("content rules = %q (%v), want %q valid", hint.ContentRules, hint.ContentRulesState, rules) + } + if hint.ExchangeState != helpers.HintValid || hint.Exchange != "exchange.test" { + t.Errorf("exchange = %q (%v), want exchange.test valid", hint.Exchange, hint.ExchangeState) + } + }) + + t.Run("a redirect carries nothing", func(t *testing.T) { + edge := refusalAnswer(t, http.StatusFound, "text/plain", "", + fieldLine{"Location", "https://elsewhere.test/doc?token=" + headerProbe}, + fieldLine{"X-Fora-Probe", headerProbe}) + cerr := fetch(t, edge) + if cerr.Header != nil { + t.Errorf("Header = %v, want nil", cerr.Header) + } + }) +} diff --git a/sdk/go/connect/strict.go b/sdk/go/connect/strict.go index 7f180977..f75ea9b9 100644 --- a/sdk/go/connect/strict.go +++ b/sdk/go/connect/strict.go @@ -28,8 +28,9 @@ import ( // envelope (its members, a known Connect code, well-formed details) and every // ErrorDetail it carries, both the binary value and the debug projection, against // the same two checks. An envelope that fails is refused as CallMalformed, and -// CallError.Code still holds the Connect code the peer answered with; the detail is -// not reported, because it is part of what was refused. A body that is empty or not +// CallError.Code, Status and Header still hold the peer's Connect code, HTTP status +// and header fields; the detail is not reported, because it is part of what was +// refused. A body that is empty or not // JSON is a gateway's answer rather than an envelope, and its status classifies it // as before. Without this option an error answer is read leniently, as connect-go // reads it. diff --git a/sdk/go/connect/strict_envelope.go b/sdk/go/connect/strict_envelope.go index a2cd465e..93be56c4 100644 --- a/sdk/go/connect/strict_envelope.go +++ b/sdk/go/connect/strict_envelope.go @@ -53,14 +53,23 @@ const errorDetailType = "fora.v1.ErrorDetail" // envelopeKey keys the capture the strict interceptor opens on a call's context. type envelopeKey struct{} -// envelopeCapture holds the error body of the response that arrived, when there was one -// and it fit under the read cap. +// envelopeCapture holds the error body of the response the current round trip received, +// when there was one and it fit under the read cap. The answer recorder forgets it before +// every round trip, as it forgets the answer, so a call an interceptor retries is judged +// on the error body of its last request alone. type envelopeCapture struct { mu sync.Mutex body []byte ok bool } +// envelopeCaptureOf returns the capture the strict interceptor opened on req's call, or +// nil when the call is not strictly decoded. +func envelopeCaptureOf(req *http.Request) *envelopeCapture { + capture, _ := req.Context().Value(envelopeKey{}).(*envelopeCapture) + return capture +} + func (c *envelopeCapture) store(body []byte) { c.mu.Lock() defer c.mu.Unlock() @@ -73,12 +82,22 @@ func (c *envelopeCapture) load() ([]byte, bool) { return c.body, c.ok } -// captureErrorBody copies an error response's body into the call's capture, when the -// strict interceptor opened one, and hands connect-go a body that reads the same bytes. -// A body past the read cap is not captured: connect-go refuses it on its own. -func captureErrorBody(req *http.Request, resp *http.Response) { - capture, ok := req.Context().Value(envelopeKey{}).(*envelopeCapture) - if !ok || resp.StatusCode < http.StatusBadRequest || resp.Body == nil { +// forget drops what an earlier round trip on the call captured. A nil capture, the one a +// call without strict decoding has, holds nothing to forget. +func (c *envelopeCapture) forget() { + if c == nil { + return + } + c.mu.Lock() + defer c.mu.Unlock() + c.body, c.ok = nil, false +} + +// captureErrorBody copies an error response's body into capture, when the strict +// interceptor opened one, and hands connect-go a body that reads the same bytes. A body +// past the read cap is not captured: connect-go refuses it on its own. +func captureErrorBody(capture *envelopeCapture, resp *http.Response) { + if capture == nil || resp.StatusCode < http.StatusBadRequest || resp.Body == nil { return } head, err := io.ReadAll(io.LimitReader(resp.Body, DefaultMaxRPCReadBytes+1)) diff --git a/sdk/go/connect/testdata/refusal-header-vectors.json b/sdk/go/connect/testdata/refusal-header-vectors.json new file mode 100644 index 00000000..9085cd86 --- /dev/null +++ b/sdk/go/connect/testdata/refusal-header-vectors.json @@ -0,0 +1,223 @@ +{ + "note": "Which failures carry the header fields of the answer that produced them, captured from the real Go client: an answer of status 400 or above on either leg, and never a redirect or a 2xx the client could not read. Each row records its own field lines only; carried maps each lowercased name to the value read, a repeated field joined with \", \", and is empty when no field is carried at all.", + "refusals": [ + { + "name": "rpc_401_accept_signature", + "leg": "rpc", + "strict": false, + "status": 401, + "content_type": "application/json", + "body": "{\"code\":\"unauthenticated\",\"message\":\"the request is not signed\"}", + "headers": [ + [ + "Accept-Signature", + "sig1=(\"@method\" \"@target-uri\" \"content-digest\" \"authorization\" \"signature-agent\";key=\"sig1\");created;expires;tag=\"web-bot-auth\"" + ] + ], + "carried": { + "accept-signature": "sig1=(\"@method\" \"@target-uri\" \"content-digest\" \"authorization\" \"signature-agent\";key=\"sig1\");created;expires;tag=\"web-bot-auth\"" + }, + "hint": { + "content_rules": "", + "content_rules_state": "absent", + "exchange": "", + "exchange_state": "absent" + } + }, + { + "name": "rpc_401_strict_envelope_refused", + "leg": "rpc", + "strict": true, + "status": 401, + "content_type": "application/json", + "body": "{\"code\":\"unauthenticated\",\"message\":\"the request is not signed\",\"bogus\":true}", + "headers": [ + [ + "Accept-Signature", + "sig1=(\"@method\" \"@target-uri\" \"content-digest\" \"authorization\" \"signature-agent\";key=\"sig1\");created;expires;tag=\"web-bot-auth\"" + ] + ], + "carried": { + "accept-signature": "sig1=(\"@method\" \"@target-uri\" \"content-digest\" \"authorization\" \"signature-agent\";key=\"sig1\");created;expires;tag=\"web-bot-auth\"" + }, + "hint": { + "content_rules": "", + "content_rules_state": "absent", + "exchange": "", + "exchange_state": "absent" + } + }, + { + "name": "rpc_503_retry_after", + "leg": "rpc", + "strict": false, + "status": 503, + "content_type": "text/plain", + "body": "draining", + "headers": [ + [ + "Retry-After", + "86400" + ] + ], + "carried": { + "retry-after": "86400" + }, + "hint": { + "content_rules": "", + "content_rules_state": "absent", + "exchange": "", + "exchange_state": "absent" + } + }, + { + "name": "rpc_400_two_field_lines", + "leg": "rpc", + "strict": false, + "status": 400, + "content_type": "application/json", + "body": "{\"code\":\"invalid_argument\",\"message\":\"refused\"}", + "headers": [ + [ + "X-Fora-Probe", + "first" + ], + [ + "X-Fora-Probe", + "second" + ] + ], + "carried": { + "x-fora-probe": "first, second" + }, + "hint": { + "content_rules": "", + "content_rules_state": "absent", + "exchange": "", + "exchange_state": "absent" + } + }, + { + "name": "rpc_302_location", + "leg": "rpc", + "strict": false, + "status": 302, + "content_type": "text/plain", + "body": "", + "headers": [ + [ + "Location", + "https://elsewhere.test/next?token=s3cret" + ], + [ + "X-Fora-Probe", + "redirect" + ] + ], + "carried": {}, + "hint": { + "content_rules": "", + "content_rules_state": "absent", + "exchange": "", + "exchange_state": "absent" + } + }, + { + "name": "rpc_200_not_json", + "leg": "rpc", + "strict": false, + "status": 200, + "content_type": "text/html", + "body": "\u003chtml\u003e\u003c/html\u003e", + "headers": [ + [ + "X-Fora-Probe", + "success" + ] + ], + "carried": {}, + "hint": { + "content_rules": "", + "content_rules_state": "absent", + "exchange": "", + "exchange_state": "absent" + } + }, + { + "name": "content_401_pop_accept_signature", + "leg": "content", + "strict": false, + "status": 401, + "content_type": "application/json", + "body": "{\"error\":\"unauthorized\",\"reason\":\"pop_sig_invalid\"}", + "headers": [ + [ + "Accept-Signature", + "sig1=(\"@method\" \"@target-uri\" \"signature-agent\";key=\"sig1\");created;expires;tag=\"web-bot-auth\"" + ] + ], + "carried": { + "accept-signature": "sig1=(\"@method\" \"@target-uri\" \"signature-agent\";key=\"sig1\");created;expires;tag=\"web-bot-auth\"" + }, + "hint": { + "content_rules": "", + "content_rules_state": "absent", + "exchange": "", + "exchange_state": "absent" + } + }, + { + "name": "content_403_discovery_hint", + "leg": "content", + "strict": false, + "status": 403, + "content_type": "application/json", + "body": "{\"error\":\"forbidden\"}", + "headers": [ + [ + "X-Content-Rules", + "https://publisher.test/.well-known/fora.json" + ], + [ + "X-FORA-Exchange", + "exchange.test" + ] + ], + "carried": { + "x-content-rules": "https://publisher.test/.well-known/fora.json", + "x-fora-exchange": "exchange.test" + }, + "hint": { + "content_rules": "https://publisher.test/.well-known/fora.json", + "content_rules_state": "valid", + "exchange": "exchange.test", + "exchange_state": "valid" + } + }, + { + "name": "content_302_location", + "leg": "content", + "strict": false, + "status": 302, + "content_type": "text/plain", + "body": "", + "headers": [ + [ + "Location", + "https://elsewhere.test/doc?token=s3cret" + ], + [ + "X-Fora-Probe", + "redirect" + ] + ], + "carried": {}, + "hint": { + "content_rules": "", + "content_rules_state": "absent", + "exchange": "", + "exchange_state": "absent" + } + } + ] +} diff --git a/sdk/go/connect/transport_failure_corpus_test.go b/sdk/go/connect/transport_failure_corpus_test.go index 68d0e19f..f2ca5789 100644 --- a/sdk/go/connect/transport_failure_corpus_test.go +++ b/sdk/go/connect/transport_failure_corpus_test.go @@ -54,10 +54,15 @@ func TestTransportFailureCorpusReplay(t *testing.T) { t.Errorf("kind %q and retryable %v disagree", want.Kind, want.Retryable) } // An answer that did not come from the service still came from a peer: its - // status is classified by its code, and that code is the failure's Code. - if got := transportFailureCode(t, i); got != want.Reason { + // status is classified by its code, and that code is the failure's Code. The + // status itself rides beside it. + callErr := transportFailureCallError(t, i) + if got := callErr.Code.String(); got != want.Reason { t.Errorf("code = %q, want %q", got, want.Reason) } + if callErr.Status != want.Status { + t.Errorf("status = %d, want %d", callErr.Status, want.Status) + } }) if want.Retryable { retryable++ @@ -72,9 +77,9 @@ func TestTransportFailureCorpusReplay(t *testing.T) { } } -// transportFailureCode replays case i through the real client and returns the -// Connect code its CallError carries. -func transportFailureCode(t *testing.T, i int) string { +// transportFailureCallError replays case i through the real client and returns its +// CallError. +func transportFailureCallError(t *testing.T, i int) *CallError { t.Helper() c := transportFailureCases[i] srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { @@ -88,5 +93,5 @@ func transportFailureCode(t *testing.T, i int) string { if !errors.As(err, &callErr) { t.Fatalf("%s: not a typed failure: %v", c.name, err) } - return callErr.Code.String() + return callErr } diff --git a/sdk/go/connect/verbs.go b/sdk/go/connect/verbs.go index 2d8ec539..905ca95a 100644 --- a/sdk/go/connect/verbs.go +++ b/sdk/go/connect/verbs.go @@ -246,6 +246,7 @@ func fetchCallError(op, signedURL string, err error) error { Kind: fetchKinds[ferr.Failure], Op: op, Status: ferr.Status, + Header: ferr.Header.Clone(), Reason: ferr.Reason, Err: err, } diff --git a/sdk/go/core/library_adoption_guard_test.go b/sdk/go/core/library_adoption_guard_test.go index 188869b1..55bcbc40 100644 --- a/sdk/go/core/library_adoption_guard_test.go +++ b/sdk/go/core/library_adoption_guard_test.go @@ -9,15 +9,14 @@ package core_test // 9421 sign/verify. If a future edit reintroduces the disease — a bespoke // sign/verify in L2 — a required marker disappears and the guard fails. // -// 2. TRANSPORT NEUTRALITY — no non-test .go file in sdk/go/core OR sdk/go/helpers -// may import connectrpc.com/*. That is the Core Invariant of the split: a -// grpc-go / plain net/http / any-transport consumer of core (and of the L1 -// helpers it builds on) compiles ZERO Connect. Connect lives ONLY in the -// bindings sdk/go/connect (client) and sdk/go/connectserver (server). This -// source-scan is the fast complement to the authoritative `go list -deps` -// closure gate (which the CI + the acceptance criteria run); the two are NOT -// equivalent (a scan cannot see a transitive import via another package), so -// both are kept — this file is the scan half for core + helpers. +// 2. TRANSPORT NEUTRALITY — no non-test .go file of a package core is built from +// (core itself, the L1 helpers it builds on, the internal packages they call +// and the generated wire types) may import connectrpc.com/*. That is the Core +// Invariant of the split: a grpc-go / plain net/http / any-transport consumer +// of core compiles ZERO Connect. Connect lives ONLY in the bindings +// sdk/go/connect (client) and sdk/go/connectserver (server). The scan follows +// imports through other packages (internal/importclosure), so a Connect import +// core would reach only through another package fails here too. // // SCOPE — deliberately the sdk/go L2/L1 packages ONLY. A tree-wide ban on // hand-rolled sign/verify would false-fire on the app's internal/httpsig + @@ -38,6 +37,8 @@ import ( "path/filepath" "strings" "testing" + + "github.com/FORA-Protocol/protocol/sdk/go/internal/importclosure" ) // evalSite is the pure guard predicate: it returns one message per violation (a @@ -121,7 +122,7 @@ func TestLibraryAdoptionGuard_Meta_PassesCleanSource(t *testing.T) { } // --------------------------------------------------------------------------- -// Transport-neutrality guard: no connectrpc.com import in core/*.go or helpers/*.go. +// Transport-neutrality guard: no connectrpc.com import in any package core is built from. // --------------------------------------------------------------------------- const connectrpcModulePrefix = "connectrpc.com" @@ -147,55 +148,50 @@ func importsConnectrpc(t *testing.T, filename, src string) (string, bool) { return "", false } -// scanNoConnectrpc asserts no non-test .go file in dir imports connectrpc.com/*. -func scanNoConnectrpc(t *testing.T, dir string) { - t.Helper() - entries, err := os.ReadDir(dir) +// TestCoreConnectrpcFree pins that no non-test .go file of a package core is built +// from imports connectrpc.com/* — the transport-neutral L2 substance, and everything +// under it, imposes no Connect dependency. The L1 helpers are among those packages, so +// this pins their neutrality from the guard that owns the core/connect boundary too; +// the helpers package also guards itself (connectrpc_free_test.go). +func TestCoreConnectrpcFree(t *testing.T) { + root := filepath.Join("..", "..", "..") + module, err := importclosure.ModulePath(root) if err != nil { - t.Fatalf("read %s: %v", dir, err) + t.Fatal(err) + } + closure, err := importclosure.Walk(root, module, module+"/sdk/go/core") + if err != nil { + t.Fatal(err) + } + for _, pkg := range []string{"sdk/go/core", "sdk/go/helpers"} { + if _, ok := closure[module+"/"+pkg]; !ok { + t.Fatalf("the closure of sdk/go/core no longer reaches %s, so the guard would not scan it", pkg) + } } scanned := 0 - for _, e := range entries { - name := e.Name() - if e.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") { - continue + for _, dir := range closure { + files, err := importclosure.SourceFiles(dir) + if err != nil { + t.Fatal(err) } - scanned++ - src := readSource(t, filepath.Join(dir, name)) - if path, bad := importsConnectrpc(t, name, src); bad { - t.Errorf( - "%s/%s imports %q; this package must be transport-neutral (zero Connect) so a non-Connect consumer compiles no connectrpc — the Connect binding lives ONLY in sdk/go/connect", - dir, name, path) + for _, file := range files { + scanned++ + name, _ := filepath.Rel(root, file) + if path, bad := importsConnectrpc(t, name, readSource(t, file)); bad { + t.Errorf( + "%s imports %q; core and every package it is built from must be transport-neutral (zero Connect) so a non-Connect consumer compiles no connectrpc — the Connect binding lives ONLY in sdk/go/connect", + name, path) + } } } if scanned == 0 { - t.Fatalf("scanned 0 non-test .go files in %s; the guard is not looking at the right directory", dir) + t.Fatal("scanned 0 non-test .go files for sdk/go/core; the guard is not looking at the right packages") } -} - -// TestCoreConnectrpcFree pins that no non-test .go file in sdk/go/core imports -// connectrpc.com/* — the transport-neutral L2 substance imposes no Connect -// dependency. -func TestCoreConnectrpcFree(t *testing.T) { - scanNoConnectrpc(t, ".") if _, statErr := os.Stat("verifier.go"); statErr != nil { t.Fatalf("expected sdk/go/core/verifier.go to exist as the guard's anchor: %v", statErr) } } -// TestHelpersConnectrpcFree pins that no non-test .go file in sdk/go/helpers imports -// connectrpc.com/* — the L1 helpers a core consumer builds on stay Connect-free too -// (the AsConnectError/ErrorDetailFrom bridge moved to sdk/go/connect). The helpers -// package also carries its own copy of this scan (connectrpc_free_test.go); this -// assertion is the core-side dual so the neutrality of BOTH layers is pinned from -// the guard that owns the core/connect boundary. -func TestHelpersConnectrpcFree(t *testing.T) { - scanNoConnectrpc(t, "../helpers") - if _, statErr := os.Stat(filepath.Join("../helpers", "errordetail.go")); statErr != nil { - t.Fatalf("expected sdk/go/helpers/errordetail.go to exist as the guard's anchor: %v", statErr) - } -} - // --- Meta-tests for the neutrality predicate: prove it CATCHES a real connectrpc // import (positive) and does NOT fire on a comment-only mention or a clean file // (negative). Parser-based, so the doc-comment false-positive class a raw substring diff --git a/sdk/go/helpers/acceptance.go b/sdk/go/helpers/acceptance.go index 3eaa9dfa..3831b134 100644 --- a/sdk/go/helpers/acceptance.go +++ b/sdk/go/helpers/acceptance.go @@ -8,6 +8,7 @@ import ( "fmt" forav1 "github.com/FORA-Protocol/protocol/gen/go/fora/v1" + "github.com/FORA-Protocol/protocol/sdk/go/internal/canonicalsign" ) // Agent offer-acceptance (fora.proto AgentAcceptance). An agent @@ -99,25 +100,7 @@ func checkAcceptanceRequester(id, domain string) error { // agent's consent to. Because signing and verifying both go through this function, // neither SignOfferAcceptance nor VerifyOfferAcceptance accepts such bytes. func CanonicalAcceptanceBytes(offer *forav1.Offer, requester *forav1.Requester, idempotencyKey string) ([]byte, error) { - if offer == nil { - return nil, errors.New("helpers: offer is nil") - } - if requester == nil { - return nil, errors.New("helpers: requester is nil") - } - if offer.GetSignature() == "" { - return nil, errors.New("helpers: cannot accept an unsigned offer (empty offer signature)") - } - if err := checkAcceptanceRequester(requester.GetId(), requester.GetDomain()); err != nil { - return nil, err - } - payload := &forav1.AgentAcceptancePayload{ - OfferSig: offer.GetSignature(), - RequesterId: requester.GetId(), - RequesterDomain: requester.GetDomain(), - IdempotencyKey: idempotencyKey, - } - return canonicalSignPayload(payload) + return canonicalsign.AcceptanceBytes(offer, requester, idempotencyKey, checkAcceptanceRequester) } // SignOfferAcceptance signs the canonical acceptance payload for the accepted @@ -125,14 +108,7 @@ func CanonicalAcceptanceBytes(offer *forav1.Offer, requester *forav1.Requester, // AgentAcceptance.signature field. requester and idempotencyKey come from the // enclosing execute request. func SignOfferAcceptance(priv ed25519.PrivateKey, offer *forav1.Offer, requester *forav1.Requester, idempotencyKey string) (string, error) { - if len(priv) != ed25519.PrivateKeySize { - return "", fmt.Errorf("helpers: ed25519 private key must be %d bytes, got %d", ed25519.PrivateKeySize, len(priv)) - } - payload, err := CanonicalAcceptanceBytes(offer, requester, idempotencyKey) - if err != nil { - return "", err - } - return hex.EncodeToString(ed25519.Sign(priv, payload)), nil + return canonicalsign.SignOfferAcceptance(priv, offer, requester, idempotencyKey, checkAcceptanceRequester) } // SignOfferAcceptanceWith signs the canonical acceptance payload through an diff --git a/sdk/go/helpers/acceptance_refusals_test.go b/sdk/go/helpers/acceptance_refusals_test.go new file mode 100644 index 00000000..4288ac30 --- /dev/null +++ b/sdk/go/helpers/acceptance_refusals_test.go @@ -0,0 +1,192 @@ +package helpers_test + +// The refusals of the acceptance canonicalizers and signers, pinned: the exact text, +// the sentinel a caller matches, and which refusal wins when an input has several +// faults. The canonical renderer and the checks around it are shared with code that +// skips one of them, so the order and the wording are part of what must not move. + +import ( + "context" + "crypto/ed25519" + "errors" + "testing" + + forav1 "github.com/FORA-Protocol/protocol/gen/go/fora/v1" + "github.com/FORA-Protocol/protocol/sdk/go/helpers" +) + +// refusal is one input that must be refused, the message it must be refused with, and +// the sentinel it must match (nil when it wraps none). +type refusal struct { + name string + call func() error + message string + sentinel error +} + +func assertRefusals(t *testing.T, cases []refusal) { + t.Helper() + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + err := c.call() + if err == nil { + t.Fatal("accepted, want a refusal") + } + if err.Error() != c.message { + t.Errorf("message = %q, want %q", err.Error(), c.message) + } + if c.sentinel != nil && !errors.Is(err, c.sentinel) { + t.Errorf("%v does not match %v", err, c.sentinel) + } + }) + } +} + +// offAlgSigner is a Signer offering an algorithm the acceptances do not accept. +type offAlgSigner struct{} + +func (offAlgSigner) KeyID() string { return "k" } +func (offAlgSigner) Algorithm() string { return "ecdsa-p256-sha256" } +func (offAlgSigner) Sign(context.Context, []byte) ([]byte, error) { return nil, nil } + +var ( + signedOffer = &forav1.Offer{Signature: "sig", Exchange: "one.example"} + unsignedOffer = &forav1.Offer{Exchange: "one.example"} + named = &forav1.Requester{Id: "agent-1", Domain: "agent.example"} + noID = &forav1.Requester{Domain: "agent.example"} + noDomain = &forav1.Requester{Id: "agent-1"} +) + +const ( + msgOfferNil = "helpers: offer is nil" + msgRequesterNil = "helpers: requester is nil" + msgUnsigned = "helpers: cannot accept an unsigned offer (empty offer signature)" + msgNoID = "helpers: acceptance names an empty requester: requester id is empty" + msgNoDomain = "helpers: acceptance names an empty requester: requester domain is empty" + msgShortKey = "helpers: ed25519 private key must be 64 bytes, got 3" +) + +func TestOfferAcceptanceRefusals(t *testing.T) { + _, priv, err := ed25519.GenerateKey(nil) + if err != nil { + t.Fatal(err) + } + signer, err := helpers.NewEd25519Signer("k", priv) + if err != nil { + t.Fatal(err) + } + canonical := func(o *forav1.Offer, r *forav1.Requester) func() error { + return func() error { _, err := helpers.CanonicalAcceptanceBytes(o, r, "idem"); return err } + } + sign := func(k ed25519.PrivateKey, o *forav1.Offer, r *forav1.Requester) func() error { + return func() error { _, err := helpers.SignOfferAcceptance(k, o, r, "idem"); return err } + } + signWith := func(s helpers.Signer, o *forav1.Offer, r *forav1.Requester) func() error { + return func() error { + _, err := helpers.SignOfferAcceptanceWith(context.Background(), s, o, r, "idem") + return err + } + } + assertRefusals(t, []refusal{ + {"canonical: nil offer", canonical(nil, named), msgOfferNil, nil}, + {"canonical: nil requester", canonical(signedOffer, nil), msgRequesterNil, nil}, + {"canonical: unsigned offer", canonical(unsignedOffer, named), msgUnsigned, nil}, + {"canonical: no requester id", canonical(signedOffer, noID), msgNoID, helpers.ErrAcceptanceRequesterEmpty}, + {"canonical: no requester domain", canonical(signedOffer, noDomain), msgNoDomain, helpers.ErrAcceptanceRequesterEmpty}, + {"canonical: nil offer before an empty requester", canonical(nil, noID), msgOfferNil, nil}, + {"canonical: unsigned offer before an empty requester", canonical(unsignedOffer, noID), msgUnsigned, nil}, + {"sign: short key before anything", sign(priv[:3], nil, noID), msgShortKey, nil}, + {"sign: unsigned offer", sign(priv, unsignedOffer, named), msgUnsigned, nil}, + {"sign: no requester id", sign(priv, signedOffer, noID), msgNoID, helpers.ErrAcceptanceRequesterEmpty}, + {"sign with: nil signer", signWith(nil, signedOffer, named), "helpers: acceptance signer is nil", nil}, + { + "sign with: an algorithm other than ed25519", signWith(offAlgSigner{}, signedOffer, noID), + `helpers: unsupported alg (ed25519 only): offer acceptance requires "ed25519", signer offers "ecdsa-p256-sha256"`, + helpers.ErrUnsupportedAlgorithm, + }, + {"sign with: no requester domain", signWith(signer, signedOffer, noDomain), msgNoDomain, helpers.ErrAcceptanceRequesterEmpty}, + }) +} + +func TestRequestAcceptanceRefusals(t *testing.T) { + _, priv, err := ed25519.GenerateKey(nil) + if err != nil { + t.Fatal(err) + } + signer, err := helpers.NewEd25519Signer("k", priv) + if err != nil { + t.Fatal(err) + } + item := func(o *forav1.Offer) *forav1.TransactionItem { return &forav1.TransactionItem{Offer: o} } + req := func(r *forav1.Requester, items ...*forav1.TransactionItem) *forav1.TransactionRequest { + return &forav1.TransactionRequest{Requester: r, IdempotencyKey: "idem", Items: items} + } + payloadOf := func(q *forav1.TransactionRequest) func() error { + return func() error { _, err := helpers.RequestAcceptancePayload(q); return err } + } + payload := func(id, domain string, items ...*forav1.AgentRequestAcceptanceItem) *forav1.AgentRequestAcceptancePayload { + return &forav1.AgentRequestAcceptancePayload{ + Items: items, RequesterId: id, RequesterDomain: domain, IdempotencyKey: "idem", + } + } + accepted := func(sig, exchange string) *forav1.AgentRequestAcceptanceItem { + return &forav1.AgentRequestAcceptanceItem{OfferSig: sig, Exchange: exchange} + } + canonical := func(p *forav1.AgentRequestAcceptancePayload) func() error { + return func() error { _, err := helpers.CanonicalRequestAcceptanceBytes(p); return err } + } + tooMany := make([]*forav1.AgentRequestAcceptanceItem, helpers.MaxRequestAcceptanceItemsForTest+1) + for i := range tooMany { + tooMany[i] = accepted("sig", "one.example") + } + sign := func(k ed25519.PrivateKey, q *forav1.TransactionRequest) func() error { + return func() error { _, err := helpers.SignRequestAcceptance(k, q); return err } + } + signWith := func(s helpers.Signer, q *forav1.TransactionRequest) func() error { + return func() error { + _, err := helpers.SignRequestAcceptanceWith(context.Background(), s, q) + return err + } + } + assertRefusals(t, []refusal{ + {"payload: nil request", payloadOf(nil), "helpers: transaction request is nil", nil}, + {"payload: nil requester", payloadOf(req(nil, item(signedOffer))), msgRequesterNil, nil}, + {"payload: no requester id", payloadOf(req(noID, item(signedOffer))), msgNoID, helpers.ErrAcceptanceRequesterEmpty}, + {"payload: no requester domain", payloadOf(req(noDomain, item(signedOffer))), msgNoDomain, helpers.ErrAcceptanceRequesterEmpty}, + {"payload: no items", payloadOf(req(named)), "helpers: transaction request has no items", nil}, + {"payload: an empty requester before no items", payloadOf(req(noID)), msgNoID, helpers.ErrAcceptanceRequesterEmpty}, + {"payload: an item with no offer", payloadOf(req(named, item(nil))), "helpers: item 0 offer is nil", nil}, + {"payload: an unsigned item", payloadOf(req(named, item(unsignedOffer))), "helpers: item 0 offer is unsigned", nil}, + { + "payload: an item naming no exchange", payloadOf(req(named, item(&forav1.Offer{Signature: "sig"}))), + "helpers: item 0 offer exchange is empty", nil, + }, + {"canonical: nil payload", canonical(nil), "helpers: request-acceptance payload is nil", nil}, + {"canonical: no requester id", canonical(payload("", "agent.example", accepted("sig", "one.example"))), msgNoID, helpers.ErrAcceptanceRequesterEmpty}, + {"canonical: no requester domain", canonical(payload("agent-1", "", accepted("sig", "one.example"))), msgNoDomain, helpers.ErrAcceptanceRequesterEmpty}, + {"canonical: no items", canonical(payload("agent-1", "agent.example")), "helpers: request-acceptance payload has no items", nil}, + {"canonical: an empty requester before no items", canonical(payload("", "agent.example")), msgNoID, helpers.ErrAcceptanceRequesterEmpty}, + { + "canonical: too many items", canonical(payload("agent-1", "agent.example", tooMany...)), + "helpers: request-acceptance payload has 257 items, the maximum is 256", nil, + }, + { + "canonical: an item with no offer signature", canonical(payload("agent-1", "agent.example", accepted("", "one.example"))), + "helpers: request-acceptance item 0 offer signature is empty", nil, + }, + { + "canonical: an item naming no exchange", canonical(payload("agent-1", "agent.example", accepted("sig", ""))), + "helpers: request-acceptance item 0 exchange is empty", nil, + }, + {"sign: short key before anything", sign(priv[:3], nil), msgShortKey, nil}, + {"sign: no requester id", sign(priv, req(noID, item(signedOffer))), msgNoID, helpers.ErrAcceptanceRequesterEmpty}, + {"sign: no items", sign(priv, req(named)), "helpers: transaction request has no items", nil}, + {"sign with: nil signer", signWith(nil, req(named, item(signedOffer))), "helpers: request-acceptance signer is nil", nil}, + { + "sign with: an algorithm other than ed25519", signWith(offAlgSigner{}, req(noID)), + `helpers: unsupported alg (ed25519 only): request acceptance requires "ed25519", signer offers "ecdsa-p256-sha256"`, + helpers.ErrUnsupportedAlgorithm, + }, + {"sign with: no requester domain", signWith(signer, req(noDomain, item(signedOffer))), msgNoDomain, helpers.ErrAcceptanceRequesterEmpty}, + }) +} diff --git a/sdk/go/helpers/canonicalsign.go b/sdk/go/helpers/canonicalsign.go index 0301603c..c38a9134 100644 --- a/sdk/go/helpers/canonicalsign.go +++ b/sdk/go/helpers/canonicalsign.go @@ -1,53 +1,12 @@ package helpers -import ( - "errors" - "fmt" +import "github.com/FORA-Protocol/protocol/sdk/go/internal/canonicalsign" - "github.com/gowebpki/jcs" - "google.golang.org/protobuf/encoding/protojson" - "google.golang.org/protobuf/proto" - "google.golang.org/protobuf/reflect/protoreflect" -) - -// Canonical signing payload (ADR-020 §4, fora.proto "canonical signing" note). -// -// Both signed FORA payloads — the Offer signature and the agent's detached -// offer-acceptance — cover a canonical serialization of a protobuf message. As of -// the JCS switch that canonical form is: -// -// signed_payload = JCS( protojson(msg with sig/sig_algorithm cleared) ) -// -// i.e. render the message to canonical proto-JSON with a PINNED option set, then -// apply the RFC 8785 JSON Canonicalization Scheme. Deterministic protobuf binary -// marshaling is explicitly NOT canonical across languages/versions (protobuf's own -// caveat), so it cannot be a cross-language signing primitive; JCS over proto-JSON -// CAN be reproduced by any language (Go/TS/Python) without a protobuf binary codec. -// Go keeps protobuf and the wire format unchanged — ONLY the signed bytes change. -// -// PINNED proto-JSON option set (documented in proto/fora.proto): -// - enums as NAME strings (protojson default; UseEnumNumbers=false) -// - int64/uint64/fixed64 as strings (protojson default) -// - bytes as std base64 (protojson default) -// - google.protobuf.Timestamp / Duration per proto-JSON WKT rules -// - OMIT unpopulated fields (EmitUnpopulated=false — protojson default) -// - field naming: snake_case (UseProtoNames=true — the proto field name, matching -// the wire, the generated Pydantic/Zod clients, and the committed corpus) -// - google.protobuf.Struct ext → a plain JSON object; JCS then sorts its keys -// recursively, so the Struct case needs no special handling. -// -// The Go-emitted golden vector is the arbiter: whatever these options render MUST -// be byte-identical across Go(protojson+jcs) / TS / Python, proven by the shared -// offer-verify and acceptance vectors. - -// canonicalSignJSONOptions is the single, pinned protojson marshal option set every -// canonical-signing site uses. Pinned here (not per call site) so the two signed -// payloads cannot drift from each other. -var canonicalSignJSONOptions = protojson.MarshalOptions{ - UseProtoNames: true, // snake_case — the naming all three SDK targets share - UseEnumNumbers: false, // enums as NAME strings - EmitUnpopulated: false, // omit unpopulated fields -} +// The canonical signing form — RFC 8785 JCS over proto-JSON under one pinned option +// set, defined on Offer.signature in fora.proto — is rendered by +// internal/canonicalsign, whose package doc states the pinned options. This +// package's offer and acceptance functions render through it, and so do the test +// signers in helperstest. // ErrUnknownFields refuses a message the canonical form cannot faithfully // represent. proto-JSON renders only fields the schema defines, so a message @@ -62,85 +21,4 @@ var canonicalSignJSONOptions = protojson.MarshalOptions{ // evidence can tell this refusal apart from a marshal fault. VerifyOffer wraps it // alongside ErrOfferSignatureInvalid — a message that arrived carrying extra bytes // is a tampered offer — so errors.Is matches either sentinel there. -var ErrUnknownFields = errors.New("helpers: message carries unknown fields; canonical bytes cannot represent them") - -// hasUnknownFields reports whether msg, or anything reachable from it, carries -// fields this build's schema does not define. -// -// The walk is explicit rather than delegated to a traversal helper: this is a -// signature-coverage guard, so its reachability rules are stated here where they -// can be audited against the message tree. Unknown-field sets are PER MESSAGE — -// a nested message and every element of a repeated or map field owns its own — -// so a top-level check alone would pass a payload whose tampering sits one level -// down. -func hasUnknownFields(msg proto.Message) bool { - if msg == nil { - return false - } - return messageHasUnknown(msg.ProtoReflect()) -} - -func messageHasUnknown(m protoreflect.Message) bool { - if !m.IsValid() { - return false // a typed nil carries nothing - } - if len(m.GetUnknown()) > 0 { - return true - } - found := false - m.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool { - switch { - case fd.IsMap(): - // Map KEYS are scalars; only a message-typed value can nest one. The - // synthetic ENTRY messages are not visited: protobuf-go discards unknown - // bytes planted on an entry at unmarshal, so they never reach here (nor - // the wire again). offer_test.go pins that, and fails loudly if a - // protobuf upgrade starts retaining them. - if fd.MapValue().Kind() == protoreflect.MessageKind || - fd.MapValue().Kind() == protoreflect.GroupKind { - v.Map().Range(func(_ protoreflect.MapKey, mv protoreflect.Value) bool { - if messageHasUnknown(mv.Message()) { - found = true - } - return !found - }) - } - case fd.IsList(): - if fd.Kind() == protoreflect.MessageKind || fd.Kind() == protoreflect.GroupKind { - list := v.List() - for i := 0; i < list.Len() && !found; i++ { - if messageHasUnknown(list.Get(i).Message()) { - found = true - } - } - } - case fd.Kind() == protoreflect.MessageKind, fd.Kind() == protoreflect.GroupKind: - if messageHasUnknown(v.Message()) { - found = true - } - } - return !found // stop the range as soon as one is found - }) - return found -} - -// canonicalSignPayload renders msg to canonical proto-JSON (pinned options) and -// applies RFC 8785 JCS, returning the exact UTF-8 bytes an Ed25519 signature -// covers. Callers clear signature/signature_algorithm on a clone BEFORE calling. -// -// A message carrying unknown fields is refused rather than rendered — see -// ErrUnknownFields. -func canonicalSignPayload(msg proto.Message) ([]byte, error) { - if hasUnknownFields(msg) { - return nil, ErrUnknownFields - } - pj, err := canonicalSignJSONOptions.Marshal(msg) - if err != nil { - return nil, fmt.Errorf("helpers: canonical proto-JSON marshal: %w", err) - } - canon, err := jcs.Transform(pj) - if err != nil { - return nil, fmt.Errorf("helpers: JCS transform: %w", err) - } - return canon, nil -} +var ErrUnknownFields = canonicalsign.ErrUnknownFields diff --git a/sdk/go/helpers/connectrpc_free_test.go b/sdk/go/helpers/connectrpc_free_test.go index 26d53485..c7f6ada1 100644 --- a/sdk/go/helpers/connectrpc_free_test.go +++ b/sdk/go/helpers/connectrpc_free_test.go @@ -2,28 +2,23 @@ package helpers_test // Structural neutrality guard for the sdk/go L1 helpers package (the // core/connect split with the folded-in AsConnectError decouple). The Core -// Invariant of that refactor is transport neutrality: a grpc-go / plain +// Invariant of that split is transport neutrality: a grpc-go / plain // net/http / any-transport consumer that only wants the neutral L1 mechanics // (ErrorDetail builders, sign/verify, key-resolution, money, signed URLs) must -// compile ZERO Connect. Today that invariant is VIOLATED — errordetail.go -// imports connectrpc.com/connect for AsConnectError(connect.Code, ...) and the -// connect.Error unwrap path in ErrorDetailFrom — so this guard FAILS now (TDD -// red). The refactor moves AsConnectError into the connect binding, at which -// point the only connectrpc importer left in helpers disappears and this guard -// passes. +// compile ZERO Connect. The Connect-coupled pieces (AsConnectError, the +// connect.Error unwrap in ErrorDetailFrom) live in the connect binding. // -// This is a STRUCTURAL guard, not a behavior test: the refactor it pins is -// behavior-preserving, so the byte-level behavior net lives in the L2 suites -// (sdk/go/core/*, sdk/go/connect/*) which must stay green through the move. This -// guard pins the package-boundary invariant those suites cannot see. +// This is a STRUCTURAL guard, not a behavior test: the byte-level behavior net +// lives in the L2 suites (sdk/go/core/*, sdk/go/connect/*). This guard pins the +// package-boundary invariant those suites cannot see. // -// SCOPE — the sdk/go/helpers directory ONLY. It is the fast source-scan -// complement to the authoritative deps-closure gate -// (`go list -deps ./sdk/go/helpers | grep -c connectrpc == 0`), which the -// IMPLEMENT step adds alongside the core-side neutrality guard. A source scan -// and the deps-closure gate are NOT equivalent (the scan cannot see a -// transitive import pulled in via another package), so both are kept; this file -// is the scan half for helpers. +// SCOPE — every package of this module that helpers is built from, helpers +// included (helpersClosure, in io_leaf_guard_test.go). A scan of this directory +// alone could not see a Connect import pulled in through another package, such +// as an internal package helpers calls; the closure follows imports through +// them. The generated wire packages under gen/ are scanned too: the check reads +// import specs, so their comments cannot trip it, and a generated Connect stub +// entering the closure would. // // Import detection uses go/parser (real import specs), NOT a raw substring // match, so a doc comment that merely mentions "connectrpc.com" (e.g. @@ -37,6 +32,8 @@ import ( "path/filepath" "strings" "testing" + + "github.com/FORA-Protocol/protocol/sdk/go/internal/importclosure" ) const connectrpcModulePrefix = "connectrpc.com" @@ -74,31 +71,28 @@ func readSourceFile(t *testing.T, path string) string { return string(b) } -// TestHelpersConnectrpcFree pins the neutrality invariant: no non-test .go file -// in sdk/go/helpers may import connectrpc.com/*. It FAILS today because -// errordetail.go imports connectrpc.com/connect; it passes once the refactor -// moves AsConnectError into the connect binding. +// TestHelpersConnectrpcFree pins the neutrality invariant: no non-test .go file of +// a package helpers is built from may import connectrpc.com/*. func TestHelpersConnectrpcFree(t *testing.T) { - entries, err := os.ReadDir(".") - if err != nil { - t.Fatalf("read helpers dir: %v", err) - } + root, _, closure := helpersClosure(t) scanned := 0 - for _, e := range entries { - name := e.Name() - if e.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") { - continue + for _, dir := range closure { + files, err := importclosure.SourceFiles(dir) + if err != nil { + t.Fatal(err) } - scanned++ - src := readSourceFile(t, name) - if path, bad := importsConnectrpc(t, name, src); bad { - t.Errorf( - "sdk/go/helpers/%s imports %q; the L1 helpers package must be transport-neutral (zero Connect) so a non-Connect consumer compiles no connectrpc — move the Connect-coupled code (AsConnectError / connect.Error unwrap) into the connect binding", - name, path) + for _, file := range files { + scanned++ + name, _ := filepath.Rel(root, file) + if path, bad := importsConnectrpc(t, name, readSourceFile(t, file)); bad { + t.Errorf( + "%s imports %q; every package the L1 helpers are built from must be transport-neutral (zero Connect) so a non-Connect consumer compiles no connectrpc — move the Connect-coupled code (AsConnectError / connect.Error unwrap) into the connect binding", + name, path) + } } } if scanned == 0 { - t.Fatal("scanned 0 non-test .go files in sdk/go/helpers; the guard is not looking at the right directory") + t.Fatal("scanned 0 non-test .go files for sdk/go/helpers; the guard is not looking at the right packages") } // Cross-check the scan is anchored on the real package, not an empty cwd. if _, statErr := os.Stat(filepath.Join(".", "errordetail.go")); statErr != nil { diff --git a/sdk/go/helpers/export_test.go b/sdk/go/helpers/export_test.go index a396fe56..2f1a5555 100644 --- a/sdk/go/helpers/export_test.go +++ b/sdk/go/helpers/export_test.go @@ -4,6 +4,7 @@ import ( "context" "net/http" + "github.com/FORA-Protocol/protocol/sdk/go/internal/canonicalsign" "github.com/santhosh-tekuri/jsonschema/v6" ) @@ -17,7 +18,7 @@ func RefusingSchemaLoaderForTest() jsonschema.URLLoader { return refusingSchemaL // external test binary can pin it to the wire rule on // AgentRequestAcceptancePayload.items. Declared in a _test.go file, so it never // reaches the API-parity gate. -const MaxRequestAcceptanceItemsForTest = maxRequestAcceptanceItems +const MaxRequestAcceptanceItemsForTest = canonicalsign.MaxRequestAcceptanceItems // RawSignature describes a signature by its exact covered set and parameters, for // the tests that need a signature the shipped signers refuse to make: no tag, a diff --git a/sdk/go/helpers/gen_errordetail_vectors_test.go b/sdk/go/helpers/gen_errordetail_vectors_test.go index 8318ac7d..1dcd0fe8 100644 --- a/sdk/go/helpers/gen_errordetail_vectors_test.go +++ b/sdk/go/helpers/gen_errordetail_vectors_test.go @@ -15,7 +15,7 @@ package helpers // carries), its typed reason is read back through the REAL helpers.Reason // accessor, and its wire form is the PINNED canonical proto-JSON (snake_case field // names, enums as NAME strings, unpopulated omitted) — the same option set the -// signing payload uses (canonicalsign.go) and the exact form the generated +// signing payload uses (internal/canonicalsign) and the exact form the generated // Pydantic/Zod ErrorDetail models parse. Deterministic protobuf BINARY is // explicitly NOT a cross-language primitive (protobuf's own caveat), so the shared // wire is proto-JSON, which any language reproduces without a binary codec. @@ -89,7 +89,7 @@ func fieldErrorProjection(d *forav1.ErrorDetail) []errorDetailFieldError { } // errorDetailJSONOptions is the PINNED proto-JSON option set the ErrorDetail wire -// form uses — identical to canonicalSignJSONOptions: snake_case field names +// form uses — identical to canonicalsign.JSONOptions: snake_case field names // (UseProtoNames), enums as NAME strings (UseEnumNumbers=false), unpopulated // omitted (EmitUnpopulated=false). This is the exact shape the generated // Pydantic/Zod ErrorDetail models parse. diff --git a/sdk/go/helpers/gen_request_acceptance_vectors_test.go b/sdk/go/helpers/gen_request_acceptance_vectors_test.go index 3f3e5148..6c353513 100644 --- a/sdk/go/helpers/gen_request_acceptance_vectors_test.go +++ b/sdk/go/helpers/gen_request_acceptance_vectors_test.go @@ -10,6 +10,7 @@ import ( "testing" forav1 "github.com/FORA-Protocol/protocol/gen/go/fora/v1" + "github.com/FORA-Protocol/protocol/sdk/go/internal/canonicalsign" ) type requestAcceptanceVectorItem struct { @@ -138,7 +139,7 @@ func buildRefusedRequestAcceptanceVectors(t *testing.T, priv ed25519.PrivateKey) OfferSig: item.OfferSig, Exchange: item.Exchange, }) } - unchecked, err := canonicalSignPayload(payload) + unchecked, err := canonicalsign.Render(payload) if err != nil { t.Fatalf("%s: unchecked render: %v", v.Name, err) } diff --git a/sdk/go/helpers/gen_vectors_test.go b/sdk/go/helpers/gen_vectors_test.go index f01ff1c0..bcf42e18 100644 --- a/sdk/go/helpers/gen_vectors_test.go +++ b/sdk/go/helpers/gen_vectors_test.go @@ -39,6 +39,7 @@ import ( "time" forav1 "github.com/FORA-Protocol/protocol/gen/go/fora/v1" + "github.com/FORA-Protocol/protocol/sdk/go/internal/canonicalsign" "github.com/FORA-Protocol/protocol/sdk/go/internal/vectorio" "github.com/gowebpki/jcs" "google.golang.org/protobuf/encoding/protojson" @@ -624,7 +625,7 @@ type offerVerifyDoc struct { // reproduce JCS(protojson(offer)) byte-for-byte. func offerCanonicalProtoJSON(t *testing.T, offer *forav1.Offer) json.RawMessage { t.Helper() - pj, err := canonicalSignJSONOptions.Marshal(offer) + pj, err := canonicalsign.JSONOptions.Marshal(offer) if err != nil { t.Fatalf("offer proto-JSON marshal: %v", err) } @@ -1758,7 +1759,7 @@ func buildAcceptanceVectors(t *testing.T) []acceptanceVector { // buildRefusedAcceptanceVectors records one acceptance per requester field left // empty, and checks the Go oracle refuses each with ErrAcceptanceRequesterEmpty at // all three entry points. The recorded bytes are rendered and signed BELOW the -// refusal (canonicalSignPayload and a raw Ed25519 signature), so a port can prove +// refusal (canonicalsign.Render and a raw Ed25519 signature), so a port can prove // its verifier refuses them although the signature over them verifies. func buildRefusedAcceptanceVectors(t *testing.T) []refusedAcceptanceVector { t.Helper() @@ -1778,7 +1779,7 @@ func buildRefusedAcceptanceVectors(t *testing.T) []refusedAcceptanceVector { v := &specs[i] offer := &forav1.Offer{Signature: v.OfferSig} requester := &forav1.Requester{Id: v.RequesterID, Domain: v.RequesterDomain} - unchecked, err := canonicalSignPayload(&forav1.AgentAcceptancePayload{ + unchecked, err := canonicalsign.Render(&forav1.AgentAcceptancePayload{ OfferSig: v.OfferSig, RequesterId: v.RequesterID, RequesterDomain: v.RequesterDomain, IdempotencyKey: v.IdempotencyKey, }) diff --git a/sdk/go/helpers/io_leaf_guard_test.go b/sdk/go/helpers/io_leaf_guard_test.go index 35ea5f18..182e5cc2 100644 --- a/sdk/go/helpers/io_leaf_guard_test.go +++ b/sdk/go/helpers/io_leaf_guard_test.go @@ -13,12 +13,23 @@ package helpers_test // pulled in http.Client / http.Get / net.Dial / an http.Transport would drag the // pre-auth-reachable network surface into the trust core — exactly the regression // this guard bans, matching the other two SDKs' io-leaf guards. +// +// The scan covers every package of this module that helpers is built from, not only +// this directory: code helpers calls in an internal package (the canonical renderer in +// internal/canonicalsign, for one) is as much the trust core as helpers' own files, and +// moving code there must not move it out of the guard's reach. The generated wire +// packages under gen/ are left out of this substring scan: they are regenerated rather +// than written, and the connect-free guard (connectrpc_free_test.go) checks their +// imports. import ( "os" "path/filepath" + "sort" "strings" "testing" + + "github.com/FORA-Protocol/protocol/sdk/go/internal/importclosure" ) // forbiddenDialingTokens are the DIAL/socket surfaces a pure-tier file must not @@ -52,34 +63,63 @@ func importsDialingSurface(source string) bool { return false } -func pureHelperSources(t *testing.T) []string { +// helpersClosure returns the module root, the module path, and the directory of every +// package of this module that sdk/go/helpers is built from, helpers included, keyed by +// import path. Both purity guards of this package scan it. +func helpersClosure(t *testing.T) (root, module string, closure map[string]string) { t.Helper() - matches, err := filepath.Glob("*.go") + root = filepath.Join("..", "..", "..") + module, err := importclosure.ModulePath(root) + if err != nil { + t.Fatal(err) + } + closure, err = importclosure.Walk(root, module, module+"/sdk/go/helpers") if err != nil { - t.Fatalf("glob helpers sources: %v", err) + t.Fatal(err) + } + // Pinned, so a closure that stopped reaching them fails here instead of scanning + // less: helpers itself, the canonical renderer it calls, and the wire types it reads. + for _, pkg := range []string{"sdk/go/helpers", "sdk/go/internal/canonicalsign", "gen/go/fora/v1"} { + if _, ok := closure[module+"/"+pkg]; !ok { + t.Fatalf("the closure of sdk/go/helpers no longer reaches %s, so the purity guards would not scan it", pkg) + } } - out := make([]string, 0, len(matches)) - for _, m := range matches { - if strings.HasSuffix(m, "_test.go") { + return root, module, closure +} + +// pureHelperSources returns the module root and every non-test Go file of a package +// helpers is built from, except the generated ones. +func pureHelperSources(t *testing.T) (string, []string) { + t.Helper() + root, module, closure := helpersClosure(t) + var out []string + for path, dir := range closure { + if strings.HasPrefix(path, module+"/gen/") { continue } - out = append(out, m) + files, err := importclosure.SourceFiles(dir) + if err != nil { + t.Fatal(err) + } + out = append(out, files...) } - return out + sort.Strings(out) + return root, out } func TestHelpersIoLeafGuard_NoDialingSurface(t *testing.T) { - sources := pureHelperSources(t) + root, sources := pureHelperSources(t) if len(sources) == 0 { t.Fatal("no helpers sources found — the io-leaf guard would vacuously pass") } - for _, name := range sources { - b, err := os.ReadFile(name) + for _, file := range sources { + b, err := os.ReadFile(file) //nolint:gosec // a source file of this module if err != nil { - t.Fatalf("read %s: %v", name, err) + t.Fatalf("read %s: %v", file, err) } if importsDialingSurface(string(b)) { - t.Errorf("pure helpers file %s names a DIALING/socket surface — the trust core must open no sockets; move any fetch into sdk/go/resolvers behind the SSRF guard", name) + name, _ := filepath.Rel(root, file) + t.Errorf("pure L1 file %s names a DIALING/socket surface — the trust core must open no sockets; move any fetch into sdk/go/resolvers behind the SSRF guard", name) } } } diff --git a/sdk/go/helpers/offer.go b/sdk/go/helpers/offer.go index 6fe1e202..b8ee2899 100644 --- a/sdk/go/helpers/offer.go +++ b/sdk/go/helpers/offer.go @@ -7,6 +7,7 @@ import ( "fmt" forav1 "github.com/FORA-Protocol/protocol/gen/go/fora/v1" + "github.com/FORA-Protocol/protocol/sdk/go/internal/canonicalsign" "google.golang.org/protobuf/proto" ) @@ -142,5 +143,5 @@ func CanonicalOfferBytes(offer *forav1.Offer) ([]byte, error) { } clone.Signature = "" clone.SignatureAlgorithm = "" - return canonicalSignPayload(clone) + return canonicalsign.Render(clone) } diff --git a/sdk/go/helpers/request_acceptance.go b/sdk/go/helpers/request_acceptance.go index 1c2ace86..3a4c0203 100644 --- a/sdk/go/helpers/request_acceptance.go +++ b/sdk/go/helpers/request_acceptance.go @@ -8,57 +8,17 @@ import ( "fmt" forav1 "github.com/FORA-Protocol/protocol/gen/go/fora/v1" + "github.com/FORA-Protocol/protocol/sdk/go/internal/canonicalsign" ) // ErrRequestAcceptanceSignatureInvalid signals that the agent did not sign the // request-acceptance payload presented by the caller. var ErrRequestAcceptanceSignatureInvalid = errors.New("helpers: request-acceptance signature invalid") -// maxRequestAcceptanceItems mirrors the repeated.max_items rule on -// AgentRequestAcceptancePayload.items. The helper enforces it itself, before -// any canonicalization work, because a verifier may run with wire validation -// off and rendering an unbounded caller-controlled list to canonical JSON is -// the expensive step. A test pins this constant to the wire rule. -const maxRequestAcceptanceItems = 256 - // RequestAcceptancePayload builds the complete ordered request-set payload an // agent signs before any Broker fan-out. func RequestAcceptancePayload(req *forav1.TransactionRequest) (*forav1.AgentRequestAcceptancePayload, error) { - if req == nil { - return nil, errors.New("helpers: transaction request is nil") - } - if req.GetRequester() == nil { - return nil, errors.New("helpers: requester is nil") - } - if err := checkAcceptanceRequester(req.GetRequester().GetId(), req.GetRequester().GetDomain()); err != nil { - return nil, err - } - if len(req.GetItems()) == 0 { - return nil, errors.New("helpers: transaction request has no items") - } - items := make([]*forav1.AgentRequestAcceptanceItem, 0, len(req.GetItems())) - for i, item := range req.GetItems() { - offer := item.GetOffer() - if offer == nil { - return nil, fmt.Errorf("helpers: item %d offer is nil", i) - } - if offer.GetSignature() == "" { - return nil, fmt.Errorf("helpers: item %d offer is unsigned", i) - } - if offer.GetExchange() == "" { - return nil, fmt.Errorf("helpers: item %d offer exchange is empty", i) - } - items = append(items, &forav1.AgentRequestAcceptanceItem{ - OfferSig: offer.GetSignature(), - Exchange: offer.GetExchange(), - }) - } - return &forav1.AgentRequestAcceptancePayload{ - Items: items, - RequesterId: req.GetRequester().GetId(), - RequesterDomain: req.GetRequester().GetDomain(), - IdempotencyKey: req.GetIdempotencyKey(), - }, nil + return canonicalsign.RequestAcceptancePayload(req, checkAcceptanceRequester) } // CanonicalRequestAcceptanceBytes returns the exact JCS(protojson(...)) bytes @@ -66,48 +26,12 @@ func RequestAcceptancePayload(req *forav1.TransactionRequest) (*forav1.AgentRequ // refuses a payload that names an empty requester (ErrAcceptanceRequesterEmpty), // so neither signing nor verification accepts such bytes. func CanonicalRequestAcceptanceBytes(payload *forav1.AgentRequestAcceptancePayload) ([]byte, error) { - if payload == nil { - return nil, errors.New("helpers: request-acceptance payload is nil") - } - if err := checkAcceptanceRequester(payload.GetRequesterId(), payload.GetRequesterDomain()); err != nil { - return nil, err - } - if len(payload.GetItems()) == 0 { - return nil, errors.New("helpers: request-acceptance payload has no items") - } - if len(payload.GetItems()) > maxRequestAcceptanceItems { - return nil, fmt.Errorf("helpers: request-acceptance payload has %d items, the maximum is %d", - len(payload.GetItems()), maxRequestAcceptanceItems) - } - for i, item := range payload.GetItems() { - if item.GetOfferSig() == "" { - return nil, fmt.Errorf("helpers: request-acceptance item %d offer signature is empty", i) - } - if item.GetExchange() == "" { - return nil, fmt.Errorf("helpers: request-acceptance item %d exchange is empty", i) - } - } - return canonicalSignPayload(payload) + return canonicalsign.RequestAcceptanceBytes(payload, checkAcceptanceRequester) } // SignRequestAcceptance signs req's complete ordered request set with priv. func SignRequestAcceptance(priv ed25519.PrivateKey, req *forav1.TransactionRequest) (*forav1.AgentRequestAcceptance, error) { - if len(priv) != ed25519.PrivateKeySize { - return nil, fmt.Errorf("helpers: ed25519 private key must be %d bytes, got %d", ed25519.PrivateKeySize, len(priv)) - } - payload, err := RequestAcceptancePayload(req) - if err != nil { - return nil, err - } - canonical, err := CanonicalRequestAcceptanceBytes(payload) - if err != nil { - return nil, err - } - return &forav1.AgentRequestAcceptance{ - Payload: payload, - Signature: hex.EncodeToString(ed25519.Sign(priv, canonical)), - SignatureAlgorithm: AcceptanceSignatureAlgorithm, - }, nil + return canonicalsign.SignRequestAcceptance(priv, req, checkAcceptanceRequester, AcceptanceSignatureAlgorithm) } // SignRequestAcceptanceWith is SignRequestAcceptance for a KMS/HSM-backed diff --git a/sdk/go/helperstest/helperstest.go b/sdk/go/helperstest/helperstest.go new file mode 100644 index 00000000..806274ec --- /dev/null +++ b/sdk/go/helperstest/helperstest.go @@ -0,0 +1,54 @@ +// Package helperstest signs agent acceptances that name an empty requester, for a +// test proving that a verifier refuses them. +// +// fora.proto requires an acceptance to name its requester: "A signer refuses to sign +// bytes that name an empty requester, and a verifier refuses an acceptance whose +// canonical bytes name one, even when the signature over those bytes verifies." The +// signers in package helpers are that signer, so they cannot produce the one input a +// verifier's refusal is tested with. The four functions here are their twins with that +// single refusal skipped. They render through the same canonical form and keep every +// other refusal, so an acceptance they sign is valid in every respect but the +// requester, and a verifier's refusal of it is attributable to that alone. The refused +// lists of the shared acceptance-vectors.json and request-acceptance-vectors.json record +// exactly what they produce. +// +// It is test support, as net/http/httptest is: production code never imports it, and +// an acceptance it signs is one a conforming party must never send. Python's +// fora_sdk.testing and TypeScript's @fora-protocol/sdk/testing are its twins. +package helperstest + +import ( + "crypto/ed25519" + + forav1 "github.com/FORA-Protocol/protocol/gen/go/fora/v1" + "github.com/FORA-Protocol/protocol/sdk/go/helpers" + "github.com/FORA-Protocol/protocol/sdk/go/internal/canonicalsign" +) + +// anyRequester admits every requester, an empty one included. It is unexported, so +// nothing outside this package can pass it to the renderer the signers share. +func anyRequester(string, string) error { return nil } + +// CanonicalAcceptanceBytesAnyRequester is helpers.CanonicalAcceptanceBytes without its +// refusal of an empty requester id or domain. +func CanonicalAcceptanceBytesAnyRequester(offer *forav1.Offer, requester *forav1.Requester, idempotencyKey string) ([]byte, error) { + return canonicalsign.AcceptanceBytes(offer, requester, idempotencyKey, anyRequester) +} + +// SignOfferAcceptanceAnyRequester is helpers.SignOfferAcceptance without its refusal of +// an empty requester id or domain. +func SignOfferAcceptanceAnyRequester(priv ed25519.PrivateKey, offer *forav1.Offer, requester *forav1.Requester, idempotencyKey string) (string, error) { + return canonicalsign.SignOfferAcceptance(priv, offer, requester, idempotencyKey, anyRequester) +} + +// CanonicalRequestAcceptanceBytesAnyRequester is helpers.CanonicalRequestAcceptanceBytes +// without its refusal of an empty requester id or domain. +func CanonicalRequestAcceptanceBytesAnyRequester(payload *forav1.AgentRequestAcceptancePayload) ([]byte, error) { + return canonicalsign.RequestAcceptanceBytes(payload, anyRequester) +} + +// SignRequestAcceptanceAnyRequester is helpers.SignRequestAcceptance without its refusal +// of an empty requester id or domain. +func SignRequestAcceptanceAnyRequester(priv ed25519.PrivateKey, req *forav1.TransactionRequest) (*forav1.AgentRequestAcceptance, error) { + return canonicalsign.SignRequestAcceptance(priv, req, anyRequester, helpers.AcceptanceSignatureAlgorithm) +} diff --git a/sdk/go/helperstest/helperstest_test.go b/sdk/go/helperstest/helperstest_test.go new file mode 100644 index 00000000..554b98cd --- /dev/null +++ b/sdk/go/helperstest/helperstest_test.go @@ -0,0 +1,335 @@ +package helperstest_test + +// The test signers reproduce what the shared corpora record for "a signer without the +// check", answer exactly as the signers they twin for every other input, and produce +// nothing a production verifier accepts. + +import ( + "crypto/ed25519" + "encoding/hex" + "encoding/json" + "errors" + "go/parser" + "go/token" + "io/fs" + "os" + "path/filepath" + "strconv" + "strings" + "testing" + + "google.golang.org/protobuf/proto" + + forav1 "github.com/FORA-Protocol/protocol/gen/go/fora/v1" + "github.com/FORA-Protocol/protocol/sdk/go/helpers" + "github.com/FORA-Protocol/protocol/sdk/go/helperstest" +) + +type acceptanceRow struct { + Name string `json:"name"` + Empty string `json:"empty"` + OfferSig string `json:"offer_sig"` + RequesterID string `json:"requester_id"` + RequesterDomain string `json:"requester_domain"` + IdempotencyKey string `json:"idempotency_key"` + CanonicalJCS string `json:"canonical_jcs"` + SignatureHex string `json:"signature_hex"` + SeedHex string `json:"seed_hex"` +} + +func (r acceptanceRow) inputs() (*forav1.Offer, *forav1.Requester) { + return &forav1.Offer{Signature: r.OfferSig}, &forav1.Requester{Id: r.RequesterID, Domain: r.RequesterDomain} +} + +type requestRow struct { + Name string `json:"name"` + Empty string `json:"empty"` + Items []struct { + OfferSig string `json:"offer_sig"` + Exchange string `json:"exchange"` + } `json:"items"` + RequesterID string `json:"requester_id"` + RequesterDomain string `json:"requester_domain"` + IdempotencyKey string `json:"idempotency_key"` + CanonicalJCS string `json:"canonical_jcs"` + SignatureHex string `json:"signature_hex"` + SeedHex string `json:"seed_hex"` +} + +func (r requestRow) request() *forav1.TransactionRequest { + req := &forav1.TransactionRequest{ + Requester: &forav1.Requester{Id: r.RequesterID, Domain: r.RequesterDomain}, + IdempotencyKey: r.IdempotencyKey, + } + for _, it := range r.Items { + req.Items = append(req.Items, &forav1.TransactionItem{ + Offer: &forav1.Offer{Signature: it.OfferSig, Exchange: it.Exchange}, + }) + } + return req +} + +func (r requestRow) payload() *forav1.AgentRequestAcceptancePayload { + p := &forav1.AgentRequestAcceptancePayload{ + RequesterId: r.RequesterID, RequesterDomain: r.RequesterDomain, IdempotencyKey: r.IdempotencyKey, + } + for _, it := range r.Items { + p.Items = append(p.Items, &forav1.AgentRequestAcceptanceItem{OfferSig: it.OfferSig, Exchange: it.Exchange}) + } + return p +} + +type corpus[T any] struct { + Vectors []T `json:"vectors"` + Refused []T `json:"refused"` +} + +func load[T any](t *testing.T, name string) corpus[T] { + t.Helper() + path := filepath.Join("..", "helpers", "testdata", name) + raw, err := os.ReadFile(path) //nolint:gosec // a committed test vector + if err != nil { + t.Fatalf("read %s: %v", path, err) + } + var c corpus[T] + if err := json.Unmarshal(raw, &c); err != nil { + t.Fatalf("parse %s: %v", path, err) + } + if len(c.Vectors) == 0 || len(c.Refused) == 0 { + t.Fatalf("%s carries %d vectors and %d refused rows; both are needed", path, len(c.Vectors), len(c.Refused)) + } + return c +} + +func keyOf(t *testing.T, seedHex string) (ed25519.PrivateKey, ed25519.PublicKey) { + t.Helper() + seed, err := hex.DecodeString(seedHex) + if err != nil { + t.Fatalf("decode seed: %v", err) + } + priv := ed25519.NewKeyFromSeed(seed) + return priv, priv.Public().(ed25519.PublicKey) +} + +func TestTheRefusedOfferAcceptancesAreReproduced(t *testing.T) { + empties := map[string]bool{} + for _, row := range load[acceptanceRow](t, "acceptance-vectors.json").Refused { + t.Run(row.Name, func(t *testing.T) { + priv, pub := keyOf(t, row.SeedHex) + offer, requester := row.inputs() + canonical, err := helperstest.CanonicalAcceptanceBytesAnyRequester(offer, requester, row.IdempotencyKey) + if err != nil { + t.Fatalf("render: %v", err) + } + if string(canonical) != row.CanonicalJCS { + t.Errorf("bytes = %s, want %s", canonical, row.CanonicalJCS) + } + sig, err := helperstest.SignOfferAcceptanceAnyRequester(priv, offer, requester, row.IdempotencyKey) + if err != nil { + t.Fatalf("sign: %v", err) + } + if sig != row.SignatureHex { + t.Errorf("signature = %s, want %s", sig, row.SignatureHex) + } + // The signature verifies over those bytes, and the protocol's own verifier + // and signer refuse them anyway: the empty requester is the one fault. + raw, _ := hex.DecodeString(sig) + if !ed25519.Verify(pub, canonical, raw) { + t.Error("the signature does not verify over the bytes it signs") + } + if err := helpers.VerifyOfferAcceptance(offer, requester, row.IdempotencyKey, sig, pub); !errors.Is(err, helpers.ErrAcceptanceRequesterEmpty) { + t.Errorf("VerifyOfferAcceptance = %v, want ErrAcceptanceRequesterEmpty", err) + } + if _, err := helpers.SignOfferAcceptance(priv, offer, requester, row.IdempotencyKey); !errors.Is(err, helpers.ErrAcceptanceRequesterEmpty) { + t.Errorf("SignOfferAcceptance = %v, want ErrAcceptanceRequesterEmpty", err) + } + }) + empties[row.Empty] = true + } + if !empties["requester_id"] || !empties["requester_domain"] { + t.Errorf("the refused rows cover %v; both requester fields are needed", empties) + } +} + +func TestTheRefusedRequestAcceptancesAreReproduced(t *testing.T) { + empties := map[string]bool{} + for _, row := range load[requestRow](t, "request-acceptance-vectors.json").Refused { + t.Run(row.Name, func(t *testing.T) { + priv, pub := keyOf(t, row.SeedHex) + canonical, err := helperstest.CanonicalRequestAcceptanceBytesAnyRequester(row.payload()) + if err != nil { + t.Fatalf("render: %v", err) + } + if string(canonical) != row.CanonicalJCS { + t.Errorf("bytes = %s, want %s", canonical, row.CanonicalJCS) + } + acceptance, err := helperstest.SignRequestAcceptanceAnyRequester(priv, row.request()) + if err != nil { + t.Fatalf("sign: %v", err) + } + if acceptance.GetSignature() != row.SignatureHex { + t.Errorf("signature = %s, want %s", acceptance.GetSignature(), row.SignatureHex) + } + if !proto.Equal(acceptance.GetPayload(), row.payload()) { + t.Errorf("payload = %v, want %v", acceptance.GetPayload(), row.payload()) + } + if acceptance.GetSignatureAlgorithm() != helpers.AcceptanceSignatureAlgorithm { + t.Errorf("algorithm = %q, want %q", acceptance.GetSignatureAlgorithm(), helpers.AcceptanceSignatureAlgorithm) + } + if _, err := helpers.VerifyRequestAcceptance(row.request(), acceptance, pub); !errors.Is(err, helpers.ErrAcceptanceRequesterEmpty) { + t.Errorf("VerifyRequestAcceptance = %v, want ErrAcceptanceRequesterEmpty", err) + } + if _, err := helpers.SignRequestAcceptance(priv, row.request()); !errors.Is(err, helpers.ErrAcceptanceRequesterEmpty) { + t.Errorf("SignRequestAcceptance = %v, want ErrAcceptanceRequesterEmpty", err) + } + }) + empties[row.Empty] = true + } + if !empties["requester_id"] || !empties["requester_domain"] { + t.Errorf("the refused rows cover %v; both requester fields are needed", empties) + } +} + +// On a requester that is named, the twins are the signers: same bytes, same signature. +func TestTheTwinsMatchTheSignersOnANamedRequester(t *testing.T) { + for _, row := range load[acceptanceRow](t, "acceptance-vectors.json").Vectors { + priv, _ := keyOf(t, row.SeedHex) + offer, requester := row.inputs() + want, wantErr := helpers.SignOfferAcceptance(priv, offer, requester, row.IdempotencyKey) + got, err := helperstest.SignOfferAcceptanceAnyRequester(priv, offer, requester, row.IdempotencyKey) + if wantErr != nil || err != nil || got != want || got != row.SignatureHex { + t.Errorf("%s: twin %q (%v), signer %q (%v), corpus %q", row.Name, got, err, want, wantErr, row.SignatureHex) + } + } + for _, row := range load[requestRow](t, "request-acceptance-vectors.json").Vectors { + priv, _ := keyOf(t, row.SeedHex) + want, wantErr := helpers.SignRequestAcceptance(priv, row.request()) + got, err := helperstest.SignRequestAcceptanceAnyRequester(priv, row.request()) + if wantErr != nil || err != nil || !proto.Equal(got, want) || got.GetSignature() != row.SignatureHex { + t.Errorf("%s: twin %v (%v), signer %v (%v)", row.Name, got, err, want, wantErr) + } + } +} + +// Every refusal but the requester's is the signer's own, in text and in order. +func TestEveryOtherRefusalIsKept(t *testing.T) { + _, priv, err := ed25519.GenerateKey(nil) + if err != nil { + t.Fatal(err) + } + named := &forav1.Requester{Id: "agent-1", Domain: "agent.example"} + noID := &forav1.Requester{Domain: "agent.example"} + signed := &forav1.Offer{Signature: "sig", Exchange: "one.example"} + unsigned := &forav1.Offer{Exchange: "one.example"} + item := func(o *forav1.Offer) *forav1.TransactionItem { return &forav1.TransactionItem{Offer: o} } + req := func(r *forav1.Requester, items ...*forav1.TransactionItem) *forav1.TransactionRequest { + return &forav1.TransactionRequest{Requester: r, IdempotencyKey: "idem", Items: items} + } + payload := func(items ...*forav1.AgentRequestAcceptanceItem) *forav1.AgentRequestAcceptancePayload { + return &forav1.AgentRequestAcceptancePayload{Items: items, RequesterId: "agent-1", RequesterDomain: "agent.example"} + } + tooMany := make([]*forav1.AgentRequestAcceptanceItem, 257) + for i := range tooMany { + tooMany[i] = &forav1.AgentRequestAcceptanceItem{OfferSig: "sig", Exchange: "one.example"} + } + + offerBytes := func(o *forav1.Offer, r *forav1.Requester) (func() error, func() error) { + return func() error { _, err := helpers.CanonicalAcceptanceBytes(o, r, "idem"); return err }, + func() error { _, err := helperstest.CanonicalAcceptanceBytesAnyRequester(o, r, "idem"); return err } + } + offerSign := func(k ed25519.PrivateKey, o *forav1.Offer, r *forav1.Requester) (func() error, func() error) { + return func() error { _, err := helpers.SignOfferAcceptance(k, o, r, "idem"); return err }, + func() error { _, err := helperstest.SignOfferAcceptanceAnyRequester(k, o, r, "idem"); return err } + } + requestBytes := func(p *forav1.AgentRequestAcceptancePayload) (func() error, func() error) { + return func() error { _, err := helpers.CanonicalRequestAcceptanceBytes(p); return err }, + func() error { _, err := helperstest.CanonicalRequestAcceptanceBytesAnyRequester(p); return err } + } + requestSign := func(k ed25519.PrivateKey, q *forav1.TransactionRequest) (func() error, func() error) { + return func() error { _, err := helpers.SignRequestAcceptance(k, q); return err }, + func() error { _, err := helperstest.SignRequestAcceptanceAnyRequester(k, q); return err } + } + cases := []struct { + name string + calls func() (func() error, func() error) + wantErr string // set when the twin must differ: the next fault after the requester's + }{ + {"nil offer", func() (func() error, func() error) { return offerBytes(nil, named) }, ""}, + {"nil requester", func() (func() error, func() error) { return offerBytes(signed, nil) }, ""}, + {"unsigned offer", func() (func() error, func() error) { return offerBytes(unsigned, named) }, ""}, + {"unsigned offer and no requester id", func() (func() error, func() error) { return offerBytes(unsigned, noID) }, ""}, + {"short key", func() (func() error, func() error) { return offerSign(priv[:3], signed, named) }, ""}, + {"nil payload", func() (func() error, func() error) { return requestBytes(nil) }, ""}, + {"no items", func() (func() error, func() error) { return requestBytes(payload()) }, ""}, + {"too many items", func() (func() error, func() error) { return requestBytes(payload(tooMany...)) }, ""}, + { + "an item with no signature", func() (func() error, func() error) { + return requestBytes(payload(&forav1.AgentRequestAcceptanceItem{Exchange: "one.example"})) + }, "", + }, + { + "an item naming no exchange", func() (func() error, func() error) { + return requestBytes(payload(&forav1.AgentRequestAcceptanceItem{OfferSig: "sig"})) + }, "", + }, + {"request: short key", func() (func() error, func() error) { return requestSign(priv[:3], req(named, item(signed))) }, ""}, + {"request: nil request", func() (func() error, func() error) { return requestSign(priv, nil) }, ""}, + {"request: no items", func() (func() error, func() error) { return requestSign(priv, req(named)) }, ""}, + {"request: an unsigned item", func() (func() error, func() error) { return requestSign(priv, req(named, item(unsigned))) }, ""}, + { + "request: no requester id and no items", + func() (func() error, func() error) { return requestSign(priv, req(noID)) }, + "helpers: transaction request has no items", + }, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + signer, twin := c.calls() + signerErr, twinErr := signer(), twin() + if signerErr == nil || twinErr == nil { + t.Fatalf("signer %v, twin %v; both must refuse", signerErr, twinErr) + } + want := signerErr.Error() + if c.wantErr != "" { + want = c.wantErr + } + if twinErr.Error() != want { + t.Errorf("twin = %q, want %q", twinErr.Error(), want) + } + }) + } +} + +// Production code never imports this package: what it signs, a conforming party must +// never send. +func TestNothingButTestsImportThisPackage(t *testing.T) { + const self = "github.com/FORA-Protocol/protocol/sdk/go/helperstest" + root := filepath.Join("..", "..", "..") + err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error { + if err != nil { + return err + } + if d.IsDir() { + if name := d.Name(); name == "node_modules" || name == ".git" || strings.HasPrefix(name, ".") && path != root { + return filepath.SkipDir + } + return nil + } + if !strings.HasSuffix(path, ".go") || strings.HasSuffix(path, "_test.go") { + return nil + } + file, err := parser.ParseFile(token.NewFileSet(), path, nil, parser.ImportsOnly) + if err != nil { + return err + } + for _, imp := range file.Imports { + if p, _ := strconv.Unquote(imp.Path.Value); p == self { + t.Errorf("%s imports %s", path, self) + } + } + return nil + }) + if err != nil { + t.Fatal(err) + } +} diff --git a/sdk/go/internal/canonicalsign/canonicalsign.go b/sdk/go/internal/canonicalsign/canonicalsign.go new file mode 100644 index 00000000..8c58ace3 --- /dev/null +++ b/sdk/go/internal/canonicalsign/canonicalsign.go @@ -0,0 +1,294 @@ +// Package canonicalsign renders FORA's canonical signing form and holds the +// acceptance checks around it. +// +// Both kinds of signed FORA payload — an Offer's signature and an agent's +// acceptances — cover a canonical serialization of a protobuf message, defined once +// on Offer.signature in fora.proto (ADR-020 §4): +// +// signed_payload = JCS( protojson(msg with sig/sig_algorithm cleared) ) +// +// i.e. render the message to canonical proto-JSON with a PINNED option set, then +// apply the RFC 8785 JSON Canonicalization Scheme. Deterministic protobuf binary +// marshaling is explicitly NOT canonical across languages/versions (protobuf's own +// caveat), so it cannot be a cross-language signing primitive; JCS over proto-JSON +// CAN be reproduced by any language (Go/TS/Python) without a protobuf binary codec. +// +// PINNED proto-JSON option set (documented in proto/fora.proto): +// - enums as NAME strings (protojson default; UseEnumNumbers=false) +// - int64/uint64/fixed64 as strings (protojson default) +// - bytes as std base64 (protojson default) +// - google.protobuf.Timestamp / Duration per proto-JSON WKT rules +// - OMIT unpopulated fields (EmitUnpopulated=false — protojson default) +// - field naming: snake_case (UseProtoNames=true — the proto field name, matching +// the wire, the generated Pydantic/Zod clients, and the committed corpus) +// - google.protobuf.Struct ext → a plain JSON object; JCS then sorts its keys +// recursively, so the Struct case needs no special handling. +// +// The Go-emitted golden vector is the arbiter: whatever these options render MUST +// be byte-identical across Go(protojson+jcs) / TS / Python, proven by the shared +// offer-verify and acceptance vectors. +// +// The acceptance functions take the requester check as a parameter and run it at +// the exact position the protocol's signer does, with every other check their own. +// helpers passes the check fora.proto requires of a signer; helperstest, whose test +// signers must produce what a verifier refuses, passes one that admits an empty +// requester. One renderer and one copy of every other check serve both, so neither +// can drift. +// +// The messages keep the helpers prefix: helpers' exported functions return them, and +// a caller reads them there. Internal on purpose: none of this is an API of its own. +package canonicalsign + +import ( + "crypto/ed25519" + "encoding/hex" + "errors" + "fmt" + + "github.com/gowebpki/jcs" + "google.golang.org/protobuf/encoding/protojson" + "google.golang.org/protobuf/proto" + "google.golang.org/protobuf/reflect/protoreflect" + + forav1 "github.com/FORA-Protocol/protocol/gen/go/fora/v1" +) + +// JSONOptions is the single, pinned protojson marshal option set every +// canonical-signing site uses. Pinned here (not per call site) so the signed +// payloads cannot drift from each other. +var JSONOptions = protojson.MarshalOptions{ + UseProtoNames: true, // snake_case — the naming all three SDK targets share + UseEnumNumbers: false, // enums as NAME strings + EmitUnpopulated: false, // omit unpopulated fields +} + +// ErrUnknownFields refuses a message the canonical form cannot faithfully +// represent. helpers exports it as helpers.ErrUnknownFields, which documents it. +var ErrUnknownFields = errors.New("helpers: message carries unknown fields; canonical bytes cannot represent them") + +// hasUnknownFields reports whether msg, or anything reachable from it, carries +// fields this build's schema does not define. +// +// The walk is explicit rather than delegated to a traversal helper: this is a +// signature-coverage guard, so its reachability rules are stated here where they +// can be audited against the message tree. Unknown-field sets are PER MESSAGE — +// a nested message and every element of a repeated or map field owns its own — +// so a top-level check alone would pass a payload whose tampering sits one level +// down. +func hasUnknownFields(msg proto.Message) bool { + if msg == nil { + return false + } + return messageHasUnknown(msg.ProtoReflect()) +} + +func messageHasUnknown(m protoreflect.Message) bool { + if !m.IsValid() { + return false // a typed nil carries nothing + } + if len(m.GetUnknown()) > 0 { + return true + } + found := false + m.Range(func(fd protoreflect.FieldDescriptor, v protoreflect.Value) bool { + switch { + case fd.IsMap(): + // Map KEYS are scalars; only a message-typed value can nest one. The + // synthetic ENTRY messages are not visited: protobuf-go discards unknown + // bytes planted on an entry at unmarshal, so they never reach here (nor + // the wire again). helpers/offer_test.go pins that, and fails loudly if a + // protobuf upgrade starts retaining them. + if fd.MapValue().Kind() == protoreflect.MessageKind || + fd.MapValue().Kind() == protoreflect.GroupKind { + v.Map().Range(func(_ protoreflect.MapKey, mv protoreflect.Value) bool { + if messageHasUnknown(mv.Message()) { + found = true + } + return !found + }) + } + case fd.IsList(): + if fd.Kind() == protoreflect.MessageKind || fd.Kind() == protoreflect.GroupKind { + list := v.List() + for i := 0; i < list.Len() && !found; i++ { + if messageHasUnknown(list.Get(i).Message()) { + found = true + } + } + } + case fd.Kind() == protoreflect.MessageKind, fd.Kind() == protoreflect.GroupKind: + if messageHasUnknown(v.Message()) { + found = true + } + } + return !found // stop the range as soon as one is found + }) + return found +} + +// Render renders msg to canonical proto-JSON (pinned options) and applies RFC 8785 +// JCS, returning the exact UTF-8 bytes an Ed25519 signature covers. Callers clear +// signature/signature_algorithm on a clone BEFORE calling. +// +// A message carrying unknown fields is refused rather than rendered — see +// ErrUnknownFields. +func Render(msg proto.Message) ([]byte, error) { + if hasUnknownFields(msg) { + return nil, ErrUnknownFields + } + pj, err := JSONOptions.Marshal(msg) + if err != nil { + return nil, fmt.Errorf("helpers: canonical proto-JSON marshal: %w", err) + } + canon, err := jcs.Transform(pj) + if err != nil { + return nil, fmt.Errorf("helpers: JCS transform: %w", err) + } + return canon, nil +} + +// RequesterCheck refuses a requester an acceptance may not name, given its id and +// domain. It runs after the anchor checks and before anything is rendered. +type RequesterCheck func(id, domain string) error + +// MaxRequestAcceptanceItems mirrors the repeated.max_items rule on +// AgentRequestAcceptancePayload.items. It is enforced here, before any +// canonicalization work, because a verifier may run with wire validation off and +// rendering an unbounded caller-controlled list to canonical JSON is the expensive +// step. A test pins this constant to the wire rule. +const MaxRequestAcceptanceItems = 256 + +// AcceptanceBytes returns the canonical bytes of the AgentAcceptancePayload an +// offer acceptance covers, refusing in order: a nil offer, a nil requester, an +// unsigned offer, then whatever check refuses, before rendering. +func AcceptanceBytes(offer *forav1.Offer, requester *forav1.Requester, idempotencyKey string, check RequesterCheck) ([]byte, error) { + if offer == nil { + return nil, errors.New("helpers: offer is nil") + } + if requester == nil { + return nil, errors.New("helpers: requester is nil") + } + if offer.GetSignature() == "" { + return nil, errors.New("helpers: cannot accept an unsigned offer (empty offer signature)") + } + if err := check(requester.GetId(), requester.GetDomain()); err != nil { + return nil, err + } + return Render(&forav1.AgentAcceptancePayload{ + OfferSig: offer.GetSignature(), + RequesterId: requester.GetId(), + RequesterDomain: requester.GetDomain(), + IdempotencyKey: idempotencyKey, + }) +} + +// SignOfferAcceptance signs AcceptanceBytes with priv and returns the hex +// signature, refusing a key of the wrong size before anything else. +func SignOfferAcceptance(priv ed25519.PrivateKey, offer *forav1.Offer, requester *forav1.Requester, idempotencyKey string, check RequesterCheck) (string, error) { + if err := checkKey(priv); err != nil { + return "", err + } + payload, err := AcceptanceBytes(offer, requester, idempotencyKey, check) + if err != nil { + return "", err + } + return hex.EncodeToString(ed25519.Sign(priv, payload)), nil +} + +// RequestAcceptancePayload builds the complete ordered request-set payload from +// req, refusing in order: a nil request, a nil requester, whatever check refuses, +// no items, then each item's missing offer, signature or exchange. +func RequestAcceptancePayload(req *forav1.TransactionRequest, check RequesterCheck) (*forav1.AgentRequestAcceptancePayload, error) { + if req == nil { + return nil, errors.New("helpers: transaction request is nil") + } + if req.GetRequester() == nil { + return nil, errors.New("helpers: requester is nil") + } + if err := check(req.GetRequester().GetId(), req.GetRequester().GetDomain()); err != nil { + return nil, err + } + if len(req.GetItems()) == 0 { + return nil, errors.New("helpers: transaction request has no items") + } + items := make([]*forav1.AgentRequestAcceptanceItem, 0, len(req.GetItems())) + for i, item := range req.GetItems() { + offer := item.GetOffer() + if offer == nil { + return nil, fmt.Errorf("helpers: item %d offer is nil", i) + } + if offer.GetSignature() == "" { + return nil, fmt.Errorf("helpers: item %d offer is unsigned", i) + } + if offer.GetExchange() == "" { + return nil, fmt.Errorf("helpers: item %d offer exchange is empty", i) + } + items = append(items, &forav1.AgentRequestAcceptanceItem{ + OfferSig: offer.GetSignature(), + Exchange: offer.GetExchange(), + }) + } + return &forav1.AgentRequestAcceptancePayload{ + Items: items, + RequesterId: req.GetRequester().GetId(), + RequesterDomain: req.GetRequester().GetDomain(), + IdempotencyKey: req.GetIdempotencyKey(), + }, nil +} + +// RequestAcceptanceBytes returns the canonical bytes a request acceptance covers, +// refusing in order: a nil payload, whatever check refuses, no items, more than +// MaxRequestAcceptanceItems, then each item's empty offer signature or exchange. +func RequestAcceptanceBytes(payload *forav1.AgentRequestAcceptancePayload, check RequesterCheck) ([]byte, error) { + if payload == nil { + return nil, errors.New("helpers: request-acceptance payload is nil") + } + if err := check(payload.GetRequesterId(), payload.GetRequesterDomain()); err != nil { + return nil, err + } + if len(payload.GetItems()) == 0 { + return nil, errors.New("helpers: request-acceptance payload has no items") + } + if len(payload.GetItems()) > MaxRequestAcceptanceItems { + return nil, fmt.Errorf("helpers: request-acceptance payload has %d items, the maximum is %d", + len(payload.GetItems()), MaxRequestAcceptanceItems) + } + for i, item := range payload.GetItems() { + if item.GetOfferSig() == "" { + return nil, fmt.Errorf("helpers: request-acceptance item %d offer signature is empty", i) + } + if item.GetExchange() == "" { + return nil, fmt.Errorf("helpers: request-acceptance item %d exchange is empty", i) + } + } + return Render(payload) +} + +// SignRequestAcceptance signs req's complete ordered request set with priv, +// refusing a key of the wrong size before anything else, and returns the +// acceptance carrying the payload, the hex signature and algorithm. +func SignRequestAcceptance(priv ed25519.PrivateKey, req *forav1.TransactionRequest, check RequesterCheck, algorithm string) (*forav1.AgentRequestAcceptance, error) { + if err := checkKey(priv); err != nil { + return nil, err + } + payload, err := RequestAcceptancePayload(req, check) + if err != nil { + return nil, err + } + canonical, err := RequestAcceptanceBytes(payload, check) + if err != nil { + return nil, err + } + return &forav1.AgentRequestAcceptance{ + Payload: payload, + Signature: hex.EncodeToString(ed25519.Sign(priv, canonical)), + SignatureAlgorithm: algorithm, + }, nil +} + +func checkKey(priv ed25519.PrivateKey) error { + if len(priv) != ed25519.PrivateKeySize { + return fmt.Errorf("helpers: ed25519 private key must be %d bytes, got %d", ed25519.PrivateKeySize, len(priv)) + } + return nil +} diff --git a/sdk/go/internal/failure/failure.go b/sdk/go/internal/failure/failure.go index d47973d3..eaf50033 100644 --- a/sdk/go/internal/failure/failure.go +++ b/sdk/go/internal/failure/failure.go @@ -64,6 +64,28 @@ func Name[K comparable](names map[K]string, k K) string { return "unknown" } +// RefusalHeader returns the header fields a failure carries for the answer that +// produced it: a copy of h when the peer answered with an error status, 400 or +// above, and nil otherwise. +// +// The rule has one owner because every leg must apply it identically, and the +// cases it leaves out are deliberate. A redirect carries none: its Location names +// a URL the peer chose, credentials included, and the error that refuses it +// already withholds that target, so the fields must not hand it back. A 2xx the +// client could not read is not a refusal, and a failure where no answer arrived +// has no fields to carry. +// +// The copy is the caller's to keep, since the response the fields came from is +// closed before the error surfaces. The values are the peer's: unvalidated, +// bounded only by the HTTP client's own header limit, and never rendered into an +// error message. +func RefusalHeader(status int, h http.Header) http.Header { + if status < http.StatusBadRequest { + return nil + } + return h.Clone() +} + // RefuseRedirect builds the http.Client CheckRedirect both credentialed legs // install: neither an RPC nor a bound fetch is ever legitimately redirected, and // following one would re-sign for a target the peer chose. diff --git a/sdk/go/internal/failure/failure_test.go b/sdk/go/internal/failure/failure_test.go new file mode 100644 index 00000000..061d5ce0 --- /dev/null +++ b/sdk/go/internal/failure/failure_test.go @@ -0,0 +1,40 @@ +package failure + +import ( + "net/http" + "reflect" + "testing" +) + +// RefusalHeader is the one place the rule lives: an error answer's fields are +// copied, and every other answer carries none, a redirect's Location included. +func TestRefusalHeader_CarriesOnlyAnErrorAnswersFields(t *testing.T) { + answer := http.Header{ + "Accept-Signature": {`sig1=("@method");tag="web-bot-auth"`}, + "Location": {"https://elsewhere.test/next?token=s3cret"}, + } + for _, status := range []int{0, 100, 200, 204, 301, 302, 304, 399} { + if got := RefusalHeader(status, answer); got != nil { + t.Errorf("status %d carries %v, want nil", status, got) + } + } + for _, status := range []int{400, 401, 403, 429, 500, 503, 599} { + got := RefusalHeader(status, answer) + if !reflect.DeepEqual(got, answer) { + t.Errorf("status %d carries %v, want %v", status, got, answer) + } + } + if got := RefusalHeader(http.StatusUnauthorized, nil); got != nil { + t.Errorf("no fields carries %v, want nil", got) + } +} + +// The copy is the caller's: changing it leaves the answer it came from untouched. +func TestRefusalHeader_ReturnsACopy(t *testing.T) { + answer := http.Header{"Retry-After": {"120"}} + got := RefusalHeader(http.StatusServiceUnavailable, answer) + got.Set("Retry-After", "0") + if answer.Get("Retry-After") != "120" { + t.Errorf("the copy shares storage with the answer: %v", answer) + } +} diff --git a/sdk/go/internal/importclosure/importclosure.go b/sdk/go/internal/importclosure/importclosure.go new file mode 100644 index 00000000..c5bc4efd --- /dev/null +++ b/sdk/go/internal/importclosure/importclosure.go @@ -0,0 +1,134 @@ +// Package importclosure finds every package of a module that a package is built from, +// for the guards that pin what that code may use. +// +// A guard that scans one directory pins nothing about the packages that directory +// imports: code that moves into one of them leaves the guard's reach and no test fails. +// The purity guards of sdk/go/helpers and sdk/go/core therefore scan every package of +// the module their package imports, directly or through another, as Walk computes it. +// +// It is test support, imported only by _test.go files. Like vectorio it takes no +// *testing.T and returns errors instead of failing, because a non-test package that +// imports testing registers the -test.* flags on every binary that links it. It reads +// import specs with go/parser rather than running the go command, so a guard needs +// nothing but the source tree. +package importclosure + +import ( + "bufio" + "fmt" + "go/parser" + "go/token" + "os" + "path/filepath" + "strconv" + "strings" +) + +// ModulePath reads the module path from the go.mod file in root. +func ModulePath(root string) (string, error) { + f, err := os.Open(filepath.Join(root, "go.mod")) //nolint:gosec // the module's own go.mod + if err != nil { + return "", fmt.Errorf("importclosure: %w", err) + } + defer func() { _ = f.Close() }() + lines := bufio.NewScanner(f) + for lines.Scan() { + if path, ok := strings.CutPrefix(strings.TrimSpace(lines.Text()), "module "); ok { + return strings.Trim(strings.TrimSpace(path), `"`), nil + } + } + if err := lines.Err(); err != nil { + return "", fmt.Errorf("importclosure: read %s: %w", f.Name(), err) + } + return "", fmt.Errorf("importclosure: %s names no module", f.Name()) +} + +// Walk returns the directory of start and of every package of module that start +// imports, directly or through another, keyed by import path. root is the module's root +// directory: an import path module+"/x/y" lives in root/x/y. +// +// Only the imports of non-test files count, so a package that a test alone imports is +// not part of the closure. A package of the module whose directory is missing or holds +// no non-test Go file is an error, so an import path mapped wrongly cannot shrink the +// closure without a failure. +func Walk(root, module, start string) (map[string]string, error) { + closure := map[string]string{} + pending := []string{start} + for len(pending) > 0 { + path := pending[len(pending)-1] + pending = pending[:len(pending)-1] + if _, seen := closure[path]; seen { + continue + } + dir, err := dirOf(root, module, path) + if err != nil { + return nil, err + } + files, err := SourceFiles(dir) + if err != nil { + return nil, err + } + if len(files) == 0 { + return nil, fmt.Errorf("importclosure: %s is in %s, which holds no non-test Go file", path, dir) + } + closure[path] = dir + for _, file := range files { + imports, err := importsOf(file) + if err != nil { + return nil, err + } + for _, imp := range imports { + if imp == module || strings.HasPrefix(imp, module+"/") { + pending = append(pending, imp) + } + } + } + } + return closure, nil +} + +// SourceFiles returns the paths of the non-test Go files in dir, in name order. +func SourceFiles(dir string) ([]string, error) { + entries, err := os.ReadDir(dir) + if err != nil { + return nil, fmt.Errorf("importclosure: %w", err) + } + var out []string + for _, e := range entries { + name := e.Name() + if e.IsDir() || !strings.HasSuffix(name, ".go") || strings.HasSuffix(name, "_test.go") { + continue + } + out = append(out, filepath.Join(dir, name)) + } + return out, nil +} + +// dirOf maps an import path of module to its directory under root. +func dirOf(root, module, path string) (string, error) { + if path == module { + return root, nil + } + rest, ok := strings.CutPrefix(path, module+"/") + if !ok { + return "", fmt.Errorf("importclosure: %s is not a package of %s", path, module) + } + return filepath.Join(root, filepath.FromSlash(rest)), nil +} + +// importsOf returns the import paths file names. +func importsOf(file string) ([]string, error) { + f, err := parser.ParseFile(token.NewFileSet(), file, nil, parser.ImportsOnly) + if err != nil { + return nil, fmt.Errorf("importclosure: %w", err) + } + out := make([]string, 0, len(f.Imports)) + for _, spec := range f.Imports { + path, err := strconv.Unquote(spec.Path.Value) + if err != nil { + return nil, fmt.Errorf("importclosure: %s: %w", file, err) + } + out = append(out, path) + } + return out, nil +} diff --git a/sdk/go/internal/importclosure/importclosure_test.go b/sdk/go/internal/importclosure/importclosure_test.go new file mode 100644 index 00000000..462eee04 --- /dev/null +++ b/sdk/go/internal/importclosure/importclosure_test.go @@ -0,0 +1,89 @@ +package importclosure_test + +// The walk the purity guards rely on, run on a module built for the test: it follows +// imports through other packages, ignores the standard library and imports only a test +// makes, and fails rather than returning a smaller closure when an import path does not +// map to a package. + +import ( + "maps" + "os" + "path/filepath" + "slices" + "strings" + "testing" + + "github.com/FORA-Protocol/protocol/sdk/go/internal/importclosure" +) + +const module = "example.test/m" + +// writeModule lays out files (path relative to the module root → content) under a fresh +// directory and returns it. +func writeModule(t *testing.T, files map[string]string) string { + t.Helper() + root := t.TempDir() + for name, content := range files { + path := filepath.Join(root, filepath.FromSlash(name)) + if err := os.MkdirAll(filepath.Dir(path), 0o755); err != nil { + t.Fatal(err) + } + if err := os.WriteFile(path, []byte(content), 0o600); err != nil { + t.Fatal(err) + } + } + return root +} + +func TestWalkFollowsImportsThroughOtherPackages(t *testing.T) { + root := writeModule(t, map[string]string{ + "go.mod": "module " + module + "\n\ngo 1.26\n", + "a/a.go": "package a\n\nimport (\n\t_ \"" + module + "/b\"\n\t_ \"" + module + "/c\"\n)\n", + "a/a_test.go": "package a\n\nimport _ \"" + module + "/d\"\n", + "b/b.go": "package b\n\nimport (\n\t\"fmt\"\n\n\t_ \"" + module + "/c\"\n)\n\nvar _ = fmt.Sprint\n", + "c/c.go": "package c\n", + "d/d.go": "package d\n", + }) + got, err := importclosure.Walk(root, module, module+"/a") + if err != nil { + t.Fatal(err) + } + want := []string{module + "/a", module + "/b", module + "/c"} + if paths := slices.Sorted(maps.Keys(got)); !slices.Equal(paths, want) { + t.Fatalf("closure = %v, want %v: transitive, without the standard library or a test-only import", paths, want) + } + if dir := got[module+"/c"]; dir != filepath.Join(root, "c") { + t.Errorf("%s/c is at %s, want %s", module, dir, filepath.Join(root, "c")) + } + + path, err := importclosure.ModulePath(root) + if err != nil || path != module { + t.Errorf("ModulePath = %q, %v; want %q", path, err, module) + } +} + +func TestWalkFailsOnAnImportItCannotMapToAPackage(t *testing.T) { + cases := []struct { + name string + files map[string]string + }{ + {"a missing directory", map[string]string{ + "a/a.go": "package a\n\nimport _ \"" + module + "/missing\"\n", + }}, + {"a directory with test files only", map[string]string{ + "a/a.go": "package a\n\nimport _ \"" + module + "/e\"\n", + "e/e_test.go": "package e\n", + }}, + } + for _, c := range cases { + t.Run(c.name, func(t *testing.T) { + _, err := importclosure.Walk(writeModule(t, c.files), module, module+"/a") + if err == nil { + t.Fatal("the walk returned a closure; it must fail instead of leaving the package out") + } + if !strings.HasPrefix(err.Error(), "importclosure:") { + t.Errorf("error %q does not say where it came from", err) + } + }) + } +} diff --git a/sdk/go/resolvers/contentfetch.go b/sdk/go/resolvers/contentfetch.go index 526f0ce3..4e7641de 100644 --- a/sdk/go/resolvers/contentfetch.go +++ b/sdk/go/resolvers/contentfetch.go @@ -158,9 +158,14 @@ func (f FetchFailure) String() string { return failure.Name(fetchFailureNames, f type FetchError struct { Failure FetchFailure Op string - Status int // HTTP status when the edge answered; 0 otherwise - Reason string // the edge's refusal token when it sent one - Err error + Status int // HTTP status when the edge answered; 0 otherwise + // Header holds the header fields of the edge's error answer, 400 or above: the + // Accept-Signature a refused proof is answered with, or the discovery headers + // of a 403. Nil for any other answer, a redirect's included. The values are the + // edge's, unvalidated, and Error() never renders them. + Header http.Header + Reason string // the edge's refusal token when it sent one + Err error } func (e *FetchError) Error() string { @@ -247,7 +252,8 @@ func (f *ContentFetcher) Fetch(ctx context.Context, signedURL string, signer Pro if resp.StatusCode < http.StatusOK || resp.StatusCode >= http.StatusMultipleChoices { return Content{}, &FetchError{ - Failure: FetchRefused, Op: op, Status: resp.StatusCode, Reason: edgeReason(resp.Body), + Failure: FetchRefused, Op: op, Status: resp.StatusCode, + Header: failure.RefusalHeader(resp.StatusCode, resp.Header), Reason: edgeReason(resp.Body), } } body, err := f.read(resp.Body) diff --git a/sdk/go/resolvers/contentfetch_test.go b/sdk/go/resolvers/contentfetch_test.go index cb98e1d0..193e7922 100644 --- a/sdk/go/resolvers/contentfetch_test.go +++ b/sdk/go/resolvers/contentfetch_test.go @@ -355,6 +355,51 @@ func TestContentFetcher_EdgeRefusal(t *testing.T) { } } +// A FetchError carries the header fields of the edge's error answer and of no other +// answer. A 304, or a 3xx the client cannot follow because it names no Location, +// reaches the refusal branch with its status and still carries none. +func TestContentFetcher_ARefusalCarriesItsHeaderFields(t *testing.T) { + tests := map[string]struct { + status int + carried bool + }{ + "a refused proof is asked for again": {http.StatusUnauthorized, true}, + "not modified carries none": {http.StatusNotModified, false}, + "a redirect naming no location": {http.StatusFound, false}, + } + for name, tc := range tests { + t.Run(name, func(t *testing.T) { + srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) { + w.Header().Set(helpers.AcceptSignatureHeader, helpers.PoPAcceptSignature) + w.WriteHeader(tc.status) + })) + defer srv.Close() + + _, err := plainFetcher(t, resolvers.ContentFetchOptions{}). + Fetch(context.Background(), srv.URL+"/doc", newPopSigner(t)) + var ferr *resolvers.FetchError + if !errors.As(err, &ferr) || ferr.Failure != resolvers.FetchRefused { + t.Fatalf("error = %v, want a FetchRefused FetchError", err) + } + if ferr.Status != tc.status { + t.Errorf("Status = %d, want %d", ferr.Status, tc.status) + } + if !tc.carried { + if ferr.Header != nil { + t.Errorf("Header = %v, want nil", ferr.Header) + } + return + } + if got := ferr.Header.Get(helpers.AcceptSignatureHeader); got != helpers.PoPAcceptSignature { + t.Errorf("Accept-Signature = %q, want %q", got, helpers.PoPAcceptSignature) + } + if strings.Contains(ferr.Error(), helpers.PoPAcceptSignature) { + t.Errorf("the message renders the edge's header value: %s", ferr.Error()) + } + }) + } +} + // No credential may reach a log or an error string: the signed URL carries sig, // kid, exp and agent_id in its query. func TestContentFetcher_ErrorsCarryNoCredential(t *testing.T) { diff --git a/sdk/parity/symbol-map.json b/sdk/parity/symbol-map.json index c4a00866..af753d11 100644 --- a/sdk/parity/symbol-map.json +++ b/sdk/parity/symbol-map.json @@ -1,5 +1,5 @@ { - "_comment": "Symbol-level Go->Python->TS API-surface parity map. Go is the oracle (sdk/go/{helpers,resolvers,core,connect,connectserver,profiles/music}). One entry per Go public symbol (pkg-qualified). 'python'/'ts' hold the mapped public name or null when that language exposes no public symbol of that name. allowlist_reason marks a deliberate documented divergence (shrink-only). Enforced by sdk/python/tests/test_api_surface_parity.py. TS object factories use the create* prefix and value generators use generate* (Go NewX stays idiomatic); the guarded-client names (guarded_client/guarded_async_client vs guardedFetchFromEnv) are a documented idiomatic divergence, not a transliteration.", + "_comment": "Symbol-level Go->Python->TS API-surface parity map. Go is the oracle (sdk/go/{helpers,resolvers,core,connect,connectserver,profiles/music,helperstest}). One entry per Go public symbol (pkg-qualified). 'python'/'ts' hold the mapped public name or null when that language exposes no public symbol of that name. allowlist_reason marks a deliberate documented divergence (shrink-only). Enforced by sdk/python/tests/test_api_surface_parity.py. TS object factories use the create* prefix and value generators use generate* (Go NewX stays idiomatic); the guarded-client names (guarded_client/guarded_async_client vs guardedFetchFromEnv) are a documented idiomatic divergence, not a transliteration.", "go_exclusions": { "connect.CallMalformed": "Member of the mapped connect.CallErrorKind enum. Go declares each as a package-level constant; Python spells it CallErrorKind. and TypeScript as a member of the CallErrorKind literal union, so neither is a top-level export the surface enumerator can see. The TYPE carries the parity claim and the shared corpora pin the values.", "connect.CallNotSent": "Member of the mapped connect.CallErrorKind enum. Go declares each as a package-level constant; Python spells it CallErrorKind. and TypeScript as a member of the CallErrorKind literal union, so neither is a top-level export the surface enumerator can see. The TYPE carries the parity claim and the shared corpora pin the values.", @@ -1279,6 +1279,26 @@ "allowlist_reason": null, "python": "POP_ACCEPT_SIGNATURE", "ts": "POP_ACCEPT_SIGNATURE" + }, + "helperstest.CanonicalAcceptanceBytesAnyRequester": { + "allowlist_reason": null, + "python": "jcs_acceptance_payload_any_requester", + "ts": "acceptancePayloadAnyRequester" + }, + "helperstest.SignOfferAcceptanceAnyRequester": { + "allowlist_reason": null, + "python": "sign_offer_acceptance_any_requester", + "ts": "signOfferAcceptanceAnyRequester" + }, + "helperstest.CanonicalRequestAcceptanceBytesAnyRequester": { + "allowlist_reason": null, + "python": "jcs_request_acceptance_payload_any_requester", + "ts": "requestAcceptancePayloadAnyRequester" + }, + "helperstest.SignRequestAcceptanceAnyRequester": { + "allowlist_reason": null, + "python": "sign_request_acceptance_any_requester", + "ts": "signRequestAcceptanceAnyRequester" } } } diff --git a/sdk/python/README.md b/sdk/python/README.md index 8be165d9..6c318c04 100644 --- a/sdk/python/README.md +++ b/sdk/python/README.md @@ -10,6 +10,7 @@ no I/O can be used on their own. | **L1** | **`fora_sdk`** (top level) | stateless, **IO-free** protocol mechanics — RFC 9421/7638 crypto, offer and acceptance signatures, signed URLs, validation. Includes `fora_sdk.core` (transport-neutral: `Verifier`, `VerifiedOffer`, `DiscoveryResult`), `fora_sdk.window` (`Window`) and `fora_sdk.server_verify` (the server side of RFC 9421). Byte-parity-guarded against the `sdk/go` oracle | | L2 · I/O | **`fora_sdk.resolvers`** | the only tier that dials the network: Web Bot Auth directories, well-known JWKS and `fora.json`, plus the SSRF-guarded HTTP client. Which faces take that client by default is decided by URL provenance, below — `WellKnownEndpointResolver` is the one request-derived face that still defaults to a plain client | | L2 · transport | `fora_sdk.client` (the async Connect-unary JSON client: the agent verbs **`discover` · `execute` · `report_usage` · `dispute` · `fetch`**, the account-setup verbs **`register` · `get_account_status`**, the broker verbs **`resolve` · `execute`** and the publisher verbs **`push_resources` · `remove_resources` · `refresh_catalog`**) · `fora_sdk.sync` (the same faces, blocking) | state is injected, never owned | +| test support | `fora_sdk.testing` | acceptance signers that admit an empty requester, for a test proving a verifier refuses one. Not re-exported from `fora_sdk`, and imported by no module of it | ```sh pip install fora-protocol-sdk @@ -412,6 +413,7 @@ The rest are bounds and seams with working defaults: | `validation` | `"strict"` | whether an outbound message is checked against its generated model before it is sent. Orthogonal to offer verification, which is about what comes back | | `registration_requirements` | a reader built on the guarded client, once per client | where `register` reads an Exchange's terms revision and registration schema. It holds no document cache on purpose: the contract requires the terms digest to come from a freshly fetched manifest | | `strict` | `False` | refuse an answer carrying an unknown field, or breaking a field-level or cross-field rule, and an error envelope or `ErrorDetail` the contract does not define. See [Testing a FORA service](#testing-a-fora-service) | +| `strict_requests` | `False` | refuse a plain dict, and a request carrying an unknown field or breaking a cross-field rule, before it is signed. Independent of `validation`. See [Testing a FORA service](#testing-a-fora-service) | | `before_sign` | `None` | a hook that receives each request just before it is signed. See [Testing a FORA service](#testing-a-fora-service) | **Build a client once and reuse it.** `ClientConfig`, the resolvers and the `Verifier` @@ -425,6 +427,8 @@ processes is the signing key. Requests are plain dicts in proto-JSON **snake_case**, or the generated request models from `wire.models`; a camelCase key is refused as malformed rather than silently ignored. +Any other key the model does not declare passes the check and is sent, and the server's +JSON codec discards it; `ClientConfig(strict_requests=True)` refuses it instead. `to_wire(model)` shows the JSON object a model is sent as. Responses are the generated Pydantic models. @@ -458,8 +462,9 @@ as a `CallError` carrying a `retrieval_auth_failure` detail with the edge's reas Failures arrive as one `CallError` carrying a `CallErrorKind` — `NOT_SENT`, `REFUSED`, `UNREACHABLE`, `MALFORMED`, `TOO_LARGE`, `NOT_SIGNABLE`, `UNKNOWN` — plus the peer's own -reason token, the Connect `code` when the peer answered with one, and, when the peer sent -one, a typed `ErrorDetail`, decoded from the binary `details[].value` the error carries. One failure type for +reason token, the Connect `code` when the peer answered with one, the `status` and +`headers` of a refusal, and, when the peer sent one, a typed `ErrorDetail`, decoded from +the binary `details[].value` the error carries. One failure type for every verb, so a caller branches in one place. `NOT_SENT` is worth singling out: it means the client refused before anything left the process, so retrying without changing something will fail the same way. @@ -507,7 +512,26 @@ a service through the SDK and check every answer through it: and `details`, must name a known Connect code and carry well-formed details, and every `ErrorDetail`, binary `value` and `debug` projection alike, must pass the strict `ErrorDetail` schema and the cross-field rules. A refused envelope is - `CallError(MALFORMED)` with `code` still set to the peer's Connect code. + `CallError(MALFORMED)` keeping `code`, `status` and `headers`: the peer's Connect code, + HTTP status and header fields. +- **`ClientConfig(strict_requests=True)`** refuses a request the contract does not + describe, before it is signed. A verb then takes only its generated request model or a + `RawBody`: a plain dict, or a model of another message, is `CallError(MALFORMED)` + naming the model to pass, raised before anything is resolved, read or sent. The message + as stamped, with `ver`, the configured `requester` and a purchase's reflected offers in + it, must pass the published strict JSON Schema of the request message and the + cross-field rules, and an unknown member is named by its path. It applies whatever + `validation` says, and a `RawBody` is still sent exactly as given. A keyword argument a + model does not declare is dropped by the model itself, so run a type checker over the + harness: it reports one. +- **`CallError.headers`** holds the header fields of a refusal, an answer of status 400 or + above on an RPC leg or `fetch`: a verifier's `Accept-Signature`, a gateway's + `Retry-After`, an edge's discovery headers, which + `parse_discovery_hint(err.status or 0, err.headers)` reads (`status` is `None` when no + answer arrived). It is the error's own copy, looked up case-insensitively, a field + sent on several lines joined with ", ". A refused redirect, a 2xx the client could not + read and a local or transport failure carry none, and the fields are never rendered into + the message. - **Identity helpers** in `fora_sdk.identity` mint a throwaway agent: `generate_key()`, `directory_document(keys)` for the directory to serve, and `signing_transport_for(key, directory)`. Serve the directory as @@ -515,6 +539,13 @@ a service through the SDK and check every answer through it: `sign_directory_response(authority, body, seeds, created, expires)` returns, signed by every key it lists for the authority it is served under; a verifier hands out no key that did not sign. +- **`fora_sdk.testing`** signs an acceptance that names an empty requester, which the + `fora_sdk.core` signers refuse to produce, as fora.proto requires of a signer: + `jcs_acceptance_payload_any_requester`, `sign_offer_acceptance_any_requester`, + `jcs_request_acceptance_payload_any_requester` and + `sign_request_acceptance_any_requester`. Each skips only that refusal, so the acceptance + is valid in every respect but the requester, and a verifier's refusal of it is + attributable to that alone. Test support only: import it from a test or a harness. - **Document readers** in `fora_sdk.resolvers` read what a party publishes and check it: `read_manifest(domain)`, `read_wba_directory(url_or_domain)`, `read_revocation_list(url)` and `read_license_document(license)`. Each fetches through diff --git a/sdk/python/fora_sdk/client/__init__.py b/sdk/python/fora_sdk/client/__init__.py index c778fb02..cbcf6bfb 100644 --- a/sdk/python/fora_sdk/client/__init__.py +++ b/sdk/python/fora_sdk/client/__init__.py @@ -37,6 +37,7 @@ from ._call import ( DEFAULT_CALL_TIMEOUT_SEC, DEFAULT_MAX_RPC_READ_BYTES, + Answer, BeforeSign, RawBody, Validation, @@ -176,7 +177,7 @@ def _refuse_if_closed(self, op: str) -> None: CallErrorKind.NOT_SENT, op, cause="the client has been closed" ) - async def _send(self, plan: _verbs.Plan) -> tuple[int, str]: + async def _send(self, plan: _verbs.Plan) -> Answer: """Send one RPC and read its answer under the cap. Streamed, so the cap bounds the READ. httpx buffers a whole response before @@ -200,10 +201,10 @@ async def _send(self, plan: _verbs.Plan) -> tuple[int, str]: follow_redirects=False, ) as response: refuse_unrequested_encoding(plan.op, response) - read = bounded_chunks(plan.op, plan.max_bytes, response.status_code) + read = bounded_chunks(plan.op, plan.max_bytes, response) async for chunk in response.aiter_bytes(): read.add(chunk) - return response.status_code, read.text() + return Answer(response.status_code, read.text(), response.headers) except httpx.HTTPError as exc: raise as_call_error(plan.op, exc) from exc except _ssrf.SsrfError as exc: @@ -242,9 +243,9 @@ async def discover(self, query: RequestMessage) -> DiscoveryResult: an absence. """ plan = _verbs.plan_discover(self._config, query) - status, body = await self._send(plan) + answer = await self._send(plan) return await asyncio.to_thread( - _verbs.finish_discover, self._config, plan, status, body + _verbs.finish_discover, self._config, plan, answer ) async def execute( @@ -261,8 +262,8 @@ async def execute( Checking one is the delivery edge's job: :meth:`fetch` presents the agent's proof of possession, and the edge verifies the URL's signature and binding.""" plan = _verbs.plan_execute(self._config, offer, idempotency_key) - status, body = await self._send(plan) - return _verbs.finish_execute(plan, status, body) + answer = await self._send(plan) + return _verbs.finish_execute(plan, answer) async def report_usage( self, report: RequestMessage, *, idempotency_key: str | None = None @@ -271,8 +272,8 @@ async def report_usage( plan = await asyncio.to_thread( _verbs.plan_report_usage, self._config, report, idempotency_key ) - status, body = await self._send(plan) - return _verbs.finish_report_usage(plan, status, body) + answer = await self._send(plan) + return _verbs.finish_report_usage(plan, answer) async def dispute( self, request: RequestMessage, *, idempotency_key: str | None = None @@ -281,8 +282,8 @@ async def dispute( plan = await asyncio.to_thread( _verbs.plan_dispute, self._config, request, idempotency_key ) - status, body = await self._send(plan) - return _verbs.finish_dispute(plan, status, body) + answer = await self._send(plan) + return _verbs.finish_dispute(plan, answer) async def register(self, request: RequestMessage) -> RegisterResponse: """Create this agent's account at the Exchange the request names. @@ -293,8 +294,8 @@ async def register(self, request: RequestMessage) -> RegisterResponse: requirements from a freshly fetched manifest. """ plan = await asyncio.to_thread(_verbs.plan_register, self._config, request) - status, body = await self._send(plan) - return _verbs.finish_register(plan, status, body) + answer = await self._send(plan) + return _verbs.finish_register(plan, answer) async def get_account_status( self, request: RequestMessage @@ -309,8 +310,8 @@ async def get_account_status( plan = await asyncio.to_thread( _verbs.plan_get_account_status, self._config, request ) - status, body = await self._send(plan) - return _verbs.finish_get_account_status(plan, status, body) + answer = await self._send(plan) + return _verbs.finish_get_account_status(plan, answer) async def fetch(self, signed_url: str) -> Content: """Retrieve the content a signed delivery URL names, presenting proof of @@ -357,6 +358,7 @@ async def fetch(self, signed_url: str) -> Content: op, status=response.status_code, cause="peer answered with a redirect, which this client does not follow", + headers=response.headers, ) if not response.is_success: # A refusal body is a small JSON object carrying a reason token. It is @@ -370,7 +372,7 @@ async def fetch(self, signed_url: str) -> Content: if len(refusal) >= MAX_ERROR_BODY_BYTES: break raise edge_refusal(response, refusal[:MAX_ERROR_BODY_BYTES]) - read = bounded_chunks(op, max_bytes, response.status_code) + read = bounded_chunks(op, max_bytes, response) async for chunk in response.aiter_bytes(): read.add(chunk) return read_content(url, response, read.body()) @@ -407,9 +409,9 @@ def __init__( async def resolve(self, request: RequestMessage) -> DiscoveryResult: """Run discovery through the Broker, which fans out to the Exchanges it knows.""" plan = _verbs.plan_resolve(self._config, request) - status, body = await self._send(plan) + answer = await self._send(plan) return await asyncio.to_thread( - _verbs.finish_resolve, self._config, plan, status, body + _verbs.finish_resolve, self._config, plan, answer ) async def execute( @@ -430,8 +432,8 @@ async def execute( :meth:`Client.execute` does. """ plan = _verbs.plan_broker_execute(self._config, offers, idempotency_key) - status, body = await self._send(plan) - return _verbs.finish_broker_execute(plan, status, body) + answer = await self._send(plan) + return _verbs.finish_broker_execute(plan, answer) class CatalogClient(_Face): @@ -459,20 +461,20 @@ def __init__( async def push_resources(self, request: RequestMessage) -> PushResourcesResponse: """Push or update catalog entries.""" plan = await asyncio.to_thread(_verbs.plan_push_resources, self._config, request) - status, body = await self._send(plan) - return _verbs.finish_push_resources(plan, status, body) + answer = await self._send(plan) + return _verbs.finish_push_resources(plan, answer) async def remove_resources(self, request: RequestMessage) -> RemoveResourcesResponse: """Remove the catalog entries the request's paths name.""" plan = await asyncio.to_thread(_verbs.plan_remove_resources, self._config, request) - status, body = await self._send(plan) - return _verbs.finish_remove_resources(plan, status, body) + answer = await self._send(plan) + return _verbs.finish_remove_resources(plan, answer) async def refresh_catalog(self, request: RequestMessage) -> RefreshCatalogResponse: """Ask the Exchange to refresh the tenant's catalog from its configured sources.""" plan = await asyncio.to_thread(_verbs.plan_refresh_catalog, self._config, request) - status, body = await self._send(plan) - return _verbs.finish_refresh_catalog(plan, status, body) + answer = await self._send(plan) + return _verbs.finish_refresh_catalog(plan, answer) class AdminClient(_Face): @@ -498,30 +500,30 @@ def __init__(self, config: ClientConfig, *, http: httpx.AsyncClient | None = Non async def set_tenant_fee_rate(self, request: RequestMessage) -> SetTenantFeeRateResponse: """Replace a tenant's fee rate (AdminService.SetTenantFeeRate).""" plan = _admin.plan_set_tenant_fee_rate(self._config, request) - status, body = await self._send(plan) - return _admin.finish_set_tenant_fee_rate(plan, status, body) + answer = await self._send(plan) + return _admin.finish_set_tenant_fee_rate(plan, answer) async def set_reporting_policy(self, request: RequestMessage) -> SetReportingPolicyResponse: """Replace a tenant's reporting policy (AdminService.SetReportingPolicy).""" plan = _admin.plan_set_reporting_policy(self._config, request) - status, body = await self._send(plan) - return _admin.finish_set_reporting_policy(plan, status, body) + answer = await self._send(plan) + return _admin.finish_set_reporting_policy(plan, answer) async def request_domain_verification( self, request: RequestMessage ) -> DomainVerificationChallenge: """Ask for a domain-verification challenge (ExchangeService.RequestDomainVerification).""" plan = _admin.plan_request_domain_verification(self._config, request) - status, body = await self._send(plan) - return _admin.finish_request_domain_verification(plan, status, body) + answer = await self._send(plan) + return _admin.finish_request_domain_verification(plan, answer) async def confirm_domain_verification( self, request: RequestMessage ) -> DomainVerificationResult: """Confirm a placed challenge (ExchangeService.ConfirmDomainVerification).""" plan = _admin.plan_confirm_domain_verification(self._config, request) - status, body = await self._send(plan) - return _admin.finish_confirm_domain_verification(plan, status, body) + answer = await self._send(plan) + return _admin.finish_confirm_domain_verification(plan, answer) def _fetch_inputs(config: ClientConfig, signed_url: str) -> tuple[dict[str, str], float, int]: diff --git a/sdk/python/fora_sdk/client/_admin.py b/sdk/python/fora_sdk/client/_admin.py index ab8b5d0c..c34ee14c 100644 --- a/sdk/python/fora_sdk/client/_admin.py +++ b/sdk/python/fora_sdk/client/_admin.py @@ -31,7 +31,7 @@ SetTenantFeeRateResponse, ) -from ._call import RawBody, decode, validate_request +from ._call import RawBody, validate_request from ._verbs import ( EXCHANGE_SERVICE, ClientConfig, @@ -42,11 +42,14 @@ _Route, _stamp_ver, _str_field, + decode_answer, ) if TYPE_CHECKING: from pydantic import BaseModel + from ._call import Answer + ADMIN_SERVICE = "fora.admin.v1.AdminService" @@ -57,8 +60,8 @@ def plan_set_tenant_fee_rate(cfg: ClientConfig, request: RequestMessage) -> Plan ) -def finish_set_tenant_fee_rate(plan: Plan, status: int, body: str) -> SetTenantFeeRateResponse: - return decode(plan.op, status, body, SetTenantFeeRateResponse, strict=plan.strict) # type: ignore[no-any-return] +def finish_set_tenant_fee_rate(plan: Plan, answer: Answer) -> SetTenantFeeRateResponse: + return decode_answer(plan, answer, SetTenantFeeRateResponse) # type: ignore[no-any-return] def plan_set_reporting_policy(cfg: ClientConfig, request: RequestMessage) -> Plan: @@ -68,8 +71,8 @@ def plan_set_reporting_policy(cfg: ClientConfig, request: RequestMessage) -> Pla ) -def finish_set_reporting_policy(plan: Plan, status: int, body: str) -> SetReportingPolicyResponse: - return decode(plan.op, status, body, SetReportingPolicyResponse, strict=plan.strict) # type: ignore[no-any-return] +def finish_set_reporting_policy(plan: Plan, answer: Answer) -> SetReportingPolicyResponse: + return decode_answer(plan, answer, SetReportingPolicyResponse) # type: ignore[no-any-return] def plan_request_domain_verification(cfg: ClientConfig, request: RequestMessage) -> Plan: @@ -83,10 +86,8 @@ def plan_request_domain_verification(cfg: ClientConfig, request: RequestMessage) ) -def finish_request_domain_verification( - plan: Plan, status: int, body: str -) -> DomainVerificationChallenge: - return decode(plan.op, status, body, DomainVerificationChallenge, strict=plan.strict) # type: ignore[no-any-return] +def finish_request_domain_verification(plan: Plan, answer: Answer) -> DomainVerificationChallenge: + return decode_answer(plan, answer, DomainVerificationChallenge) # type: ignore[no-any-return] def plan_confirm_domain_verification(cfg: ClientConfig, request: RequestMessage) -> Plan: @@ -100,10 +101,8 @@ def plan_confirm_domain_verification(cfg: ClientConfig, request: RequestMessage) ) -def finish_confirm_domain_verification( - plan: Plan, status: int, body: str -) -> DomainVerificationResult: - return decode(plan.op, status, body, DomainVerificationResult, strict=plan.strict) # type: ignore[no-any-return] +def finish_confirm_domain_verification(plan: Plan, answer: Answer) -> DomainVerificationResult: + return decode_answer(plan, answer, DomainVerificationResult) # type: ignore[no-any-return] def _plan_admin( @@ -117,8 +116,8 @@ def _plan_admin( route = _Route(op, cfg.base_url, ADMIN_SERVICE, method) if isinstance(request, RawBody): return _plan(cfg, route, request) - sent = _stamp_ver(op, request) - validate_request(op, sent, model, cfg.validation) + sent = _stamp_ver(op, request, model=model, strict_requests=cfg.strict_requests) + validate_request(op, sent, model, cfg.validation, strict_requests=cfg.strict_requests) return _plan(cfg, route, sent) @@ -135,7 +134,9 @@ def _plan_domain( route = _Route(op, cfg.base_url, EXCHANGE_SERVICE, method) if isinstance(request, RawBody): return _plan(cfg, route, request) - sent: dict[str, Any] = _stamp_ver(op, request) + sent: dict[str, Any] = _stamp_ver( + op, request, model=model, strict_requests=cfg.strict_requests + ) _require_recipient(op, _str_field(sent, "exchange")) - validate_request(op, sent, model, cfg.validation) + validate_request(op, sent, model, cfg.validation, strict_requests=cfg.strict_requests) return _plan(cfg, route, sent) diff --git a/sdk/python/fora_sdk/client/_call.py b/sdk/python/fora_sdk/client/_call.py index 66935a20..e5592d34 100644 --- a/sdk/python/fora_sdk/client/_call.py +++ b/sdk/python/fora_sdk/client/_call.py @@ -19,7 +19,7 @@ import json from collections.abc import Callable -from dataclasses import dataclass +from dataclasses import dataclass, field from typing import TYPE_CHECKING, Any, Literal, Protocol import httpx @@ -36,7 +36,7 @@ RequestIDHeader, ) -from ._strict import refuse_unless_strict +from ._strict import refuse_unless_strict, refuse_unless_strict_request from ._strict_envelope import check_strict_envelope from .errors import ( NOT_CANONICAL_WIRE_NAMING, @@ -48,6 +48,8 @@ ) if TYPE_CHECKING: + from collections.abc import Mapping + from pydantic import BaseModel from fora_sdk.signing_transport import SigningTransport @@ -84,6 +86,16 @@ def rpc_url(base_url: str, service: str, method: str) -> str: BeforeSign = Callable[[httpx.Request], httpx.Request] +@dataclass(frozen=True) +class Answer: + """One answer as a face's send read it: the status, the body under the read cap, and + the header fields, which a refusal carries (``CallError.headers``).""" + + status: int + body: str + headers: httpx.Headers = field(default_factory=httpx.Headers) + + @dataclass(frozen=True) class RawBody: """A request body a verb sends exactly as given — raw mode, chosen per call. @@ -251,7 +263,14 @@ def _apply_before_sign( Validation = Literal["strict", "off"] -def validate_request(op: str, message: Any, model: type[BaseModel], validation: Validation) -> None: +def validate_request( + op: str, + message: Any, + model: type[BaseModel], + validation: Validation, + *, + strict_requests: bool, +) -> None: """Refuse a request the protocol would reject anyway, before it costs a signature and a round trip. @@ -264,19 +283,31 @@ def validate_request(op: str, message: Any, model: type[BaseModel], validation: The validated value is DISCARDED and the caller's own object is what gets sent: a parse fills declared defaults, and sending those would put fields on the wire the caller never set. + + ``strict_requests`` (``ClientConfig.strict_requests``) adds the strict contract on top, + whatever ``validation`` says: the published strict JSON Schema of the request, which + refuses an unknown member at any depth and names it, and the cross-field rules. It runs + on the fully stamped message, so what it passes is what gets signed. """ - if validation == "off": - return - try: - model.model_validate(message) - except ValidationError as exc: - raise _schema_failure( - op, exc, "request failed its generated schema; the server could only refuse it" - ) from exc + if validation != "off": + try: + model.model_validate(message) + except ValidationError as exc: + raise _schema_failure( + op, exc, "request failed its generated schema; the server could only refuse it" + ) from exc + if strict_requests: + refuse_unless_strict_request(op, model, message) def decode_with_raw( - op: str, status: int, body: str, model: type[BaseModel], *, strict: bool = False + op: str, + status: int, + body: str, + model: type[BaseModel], + *, + strict: bool = False, + headers: Mapping[str, str] | None = None, ) -> tuple[Any, dict[str, Any]]: """Decode one answer and hand back the RAW object beside the parsed message. @@ -286,15 +317,21 @@ def decode_with_raw( a signature covers no such thing. Parsing the body twice would be a second read of the same bytes for no gain. """ - _refuse_redirect(op, status) - payload = _parse_json(op, status, body) - _refuse_error_answer(op, status, body, payload, strict=strict) + _refuse_redirect(op, status, headers) + payload = _parse_json(op, status, body, headers) + _refuse_error_answer(op, status, body, payload, strict=strict, headers=headers) parsed = _validate(op, payload, model, strict=strict) return parsed, payload if isinstance(payload, dict) else {} def decode( - op: str, status: int, body: str, model: type[BaseModel], *, strict: bool = False + op: str, + status: int, + body: str, + model: type[BaseModel], + *, + strict: bool = False, + headers: Mapping[str, str] | None = None, ) -> Any: """Turn one answer into a parsed message, or raise the typed failure. @@ -307,17 +344,20 @@ def decode( cross-field rule (see :func:`fora_sdk.strict.check_strict`), and an error answer whose envelope or ErrorDetail the contract does not accept (see :func:`fora_sdk.client._strict_envelope.check_strict_envelope`). + + ``headers`` are the answer's header fields. Every failure built from the answer is + handed them, and :class:`CallError` keeps them only for an error status. """ - _refuse_redirect(op, status) - payload = _parse_json(op, status, body) - _refuse_error_answer(op, status, body, payload, strict=strict) + _refuse_redirect(op, status, headers) + payload = _parse_json(op, status, body, headers) + _refuse_error_answer(op, status, body, payload, strict=strict, headers=headers) return _validate(op, payload, model, strict=strict) _HTTP_MULTIPLE_CHOICES_END = 400 -def _refuse_redirect(op: str, status: int) -> None: +def _refuse_redirect(op: str, status: int, headers: Mapping[str, str] | None = None) -> None: """A 3xx is a server that did not answer, not one that declined. Every leg refuses to follow a redirect, so one reaching a reader means the call did not @@ -332,22 +372,32 @@ def _refuse_redirect(op: str, status: int) -> None: op, status=status, cause="peer answered with a redirect, which this client does not follow", + headers=headers, ) -def _refuse_error_answer(op: str, status: int, body: str, payload: Any, *, strict: bool) -> None: +def _refuse_error_answer( + op: str, + status: int, + body: str, + payload: Any, + *, + strict: bool, + headers: Mapping[str, str] | None = None, +) -> None: """Raise the typed failure a non-2xx answer is; return on a 2xx. Under ``strict`` the envelope is checked first, and a refusal keeps the Connect code - the lenient read reports. An empty body is not an envelope — the lenient read takes it - as one naming no code — so it is classified by its status in either mode, like a body - that is not JSON, which never reaches here. + the lenient read reports, the status and the answer's header fields. An empty body is + not an envelope — the lenient read takes it as one naming no code — so it is + classified by its status in either mode, like a body that is not JSON, which never + reaches here. """ if _HTTP_OK <= status < _HTTP_MULTIPLE_CHOICES: return - error = _connect_envelope_error(op, status, payload) + error = _connect_envelope_error(op, status, payload, headers) if strict and body.strip(): - check_strict_envelope(op, status, payload, error.code) + check_strict_envelope(op, status, payload, error.code, headers=headers) raise error @@ -376,7 +426,7 @@ def _schema_failure(op: str, exc: ValidationError, summary: str) -> CallError: return malformed(op, summary) -def _parse_json(op: str, status: int, body: str) -> Any: +def _parse_json(op: str, status: int, body: str, headers: Mapping[str, str] | None = None) -> Any: # Depth BEFORE the parse, because ``json.loads`` descends recursively and aborts on a # deep document by raising ``RecursionError`` — which is not a ``ValueError``, so the # handler below never saw it, and not a failure this package says it raises. A 40 KB @@ -392,6 +442,7 @@ def _parse_json(op: str, status: int, body: str) -> Any: op, status=status, cause=f"answer nests deeper than {_MAX_BODY_DEPTH} containers", + headers=headers, ) try: return json.loads(body or "{}") @@ -411,11 +462,16 @@ def _parse_json(op: str, status: int, body: str) -> Any: reason=code, cause=exc, code=code, + headers=headers, ) from exc - raise CallError(CallErrorKind.MALFORMED, op, status=status, cause=exc) from exc + raise CallError( + CallErrorKind.MALFORMED, op, status=status, cause=exc, headers=headers + ) from exc -def _connect_envelope_error(op: str, status: int, payload: Any) -> CallError: +def _connect_envelope_error( + op: str, status: int, payload: Any, headers: Mapping[str, str] | None = None +) -> CallError: envelope = payload if isinstance(payload, dict) else {} code = envelope.get("code") code = code if isinstance(code, str) else "" @@ -441,6 +497,7 @@ def _connect_envelope_error(op: str, status: int, payload: Any) -> CallError: peer_message=detail.message if detail is not None and detail.message else "", cause=message if isinstance(message, str) else None, code=code, + headers=headers, ) diff --git a/sdk/python/fora_sdk/client/_read.py b/sdk/python/fora_sdk/client/_read.py index 29ad95c1..c6b9341b 100644 --- a/sdk/python/fora_sdk/client/_read.py +++ b/sdk/python/fora_sdk/client/_read.py @@ -74,13 +74,19 @@ def require_dialable_scheme(op: str, url: str) -> None: ) from exc -def over_cap(op: str, max_bytes: int, status: int | None = None) -> CallError: +def over_cap( + op: str, + max_bytes: int, + status: int | None = None, + headers: httpx.Headers | None = None, +) -> CallError: """The refusal for an answer past its cap.""" return CallError( CallErrorKind.TOO_LARGE, op, status=status, cause=f"body exceeds the {max_bytes} byte cap", + headers=headers, ) @@ -106,6 +112,7 @@ def refuse_unrequested_encoding(op: str, response: httpx.Response) -> None: f"peer answered with content-encoding {coding!r} after being asked for " "identity; a coding this client did not negotiate cannot be read under a bound" ), + headers=response.headers, ) @@ -130,9 +137,15 @@ def guarded_leg(plan: _verbs.Plan) -> bool: return plan.guarded -def bounded_chunks(op: str, max_bytes: int, status: int) -> _Bounded: - """An accumulator that refuses at the first byte past the cap.""" - return _Bounded(op=op, max_bytes=max_bytes, status=status) +def bounded_chunks(op: str, max_bytes: int, response: httpx.Response) -> _Bounded: + """An accumulator for ``response``'s body that refuses at the first byte past the cap. + + The refusal keeps the answer's status and header fields, as every failure built from + an answer does. + """ + return _Bounded( + op=op, max_bytes=max_bytes, status=response.status_code, headers=response.headers + ) class _Bounded: @@ -142,19 +155,22 @@ class _Bounded: and the rule must not. """ - __slots__ = ("_chunks", "_max", "_op", "_status", "_total") + __slots__ = ("_chunks", "_headers", "_max", "_op", "_status", "_total") - def __init__(self, *, op: str, max_bytes: int, status: int) -> None: + def __init__( + self, *, op: str, max_bytes: int, status: int, headers: httpx.Headers + ) -> None: self._op = op self._max = max_bytes self._status = status + self._headers = headers self._chunks: list[bytes] = [] self._total = 0 def add(self, chunk: bytes) -> None: self._total += len(chunk) if self._total > self._max: - raise over_cap(self._op, self._max, self._status) + raise over_cap(self._op, self._max, self._status, self._headers) self._chunks.append(chunk) @property diff --git a/sdk/python/fora_sdk/client/_strict.py b/sdk/python/fora_sdk/client/_strict.py index 52dd4632..6f6abbd8 100644 --- a/sdk/python/fora_sdk/client/_strict.py +++ b/sdk/python/fora_sdk/client/_strict.py @@ -1,9 +1,10 @@ -"""Strict response decoding: refuse an answer the contract does not describe. +"""Strict decoding, and strict requests: refuse a message the contract does not describe. -``ClientConfig.strict`` turns this on. The check itself is :mod:`fora_sdk.strict`, the -one the document readers and the error-envelope check use too; this module only names -the message a verb's generated model decodes and reports a refusal the way every other -answer failure is reported, as a MALFORMED ``CallError``. +``ClientConfig.strict`` turns it on for answers, ``ClientConfig.strict_requests`` for +requests. The check itself is :mod:`fora_sdk.strict`, the one the document readers and the +error-envelope check use too; this module only names the message a verb's generated model +describes and reports a refusal the way every other client failure is reported, as a +MALFORMED ``CallError``. """ from __future__ import annotations @@ -28,9 +29,21 @@ def refuse_unless_strict(op: str, model: type[BaseModel], payload: Any) -> None: complaint for an unknown field or a field-level rule, or the violated rule ids for a cross-field rule. """ + _refuse_unless_strict(op, model, payload, "strict decoding refused the answer") + + +def refuse_unless_strict_request(op: str, model: type[BaseModel], payload: Any) -> None: + """Refuse an outgoing ``model`` request the strict contract does not accept, before it + is signed (``ClientConfig.strict_requests``). The same check as an answer's, naming the + first failure the same way: an unknown member by its path, so a misspelled key cannot + reach the wire unnoticed.""" + _refuse_unless_strict(op, model, payload, "strict_requests refused the request") + + +def _refuse_unless_strict(op: str, model: type[BaseModel], payload: Any, lead: str) -> None: problem = strict_violation(_schema_name(model), payload) if problem is not None: - raise malformed(op, f"strict decoding refused the answer{problem}") + raise malformed(op, f"{lead}{problem}") @cache diff --git a/sdk/python/fora_sdk/client/_strict_envelope.py b/sdk/python/fora_sdk/client/_strict_envelope.py index 7ab1bf41..7b355e2d 100644 --- a/sdk/python/fora_sdk/client/_strict_envelope.py +++ b/sdk/python/fora_sdk/client/_strict_envelope.py @@ -23,7 +23,7 @@ from __future__ import annotations -from typing import Any +from typing import TYPE_CHECKING, Any from fora_sdk._errordetail_wire import WireDecodeError, _b64decode, decode_error_detail_value from fora_sdk.errordetail import ERROR_DETAIL_TYPE, REASON_FIELDS, _proto_names, _TooDeepError @@ -31,6 +31,9 @@ from .errors import CallError, CallErrorKind +if TYPE_CHECKING: + from collections.abc import Mapping + #: The members the Connect protocol defines for an error envelope and for a details entry. _ENVELOPE_MEMBERS = frozenset({"code", "message", "details"}) _ENTRY_MEMBERS = frozenset({"type", "value", "debug"}) @@ -58,12 +61,19 @@ ) -def check_strict_envelope(op: str, status: int, payload: Any, code: str | None) -> None: +def check_strict_envelope( + op: str, + status: int, + payload: Any, + code: str | None, + *, + headers: Mapping[str, str] | None = None, +) -> None: """Refuse an error answer whose envelope the strict contract does not accept. - Raises the MALFORMED ``CallError`` naming the first failure. It keeps ``status`` and - ``code``, the Connect code the lenient read reports, and carries no detail: the detail - is part of what was refused. + Raises the MALFORMED ``CallError`` naming the first failure. It keeps ``status``, + ``code`` (the Connect code the lenient read reports) and the answer's header fields, + and carries no detail: the detail is part of what was refused. """ problem = envelope_violation(payload) if problem is not None: @@ -73,6 +83,7 @@ def check_strict_envelope(op: str, status: int, payload: Any, code: str | None) status=status, cause=f"strict decoding refused the error answer: {problem}", code=code, + headers=headers, ) diff --git a/sdk/python/fora_sdk/client/_verbs.py b/sdk/python/fora_sdk/client/_verbs.py index ca59b16d..7b42ccb1 100644 --- a/sdk/python/fora_sdk/client/_verbs.py +++ b/sdk/python/fora_sdk/client/_verbs.py @@ -93,6 +93,8 @@ from fora_sdk.signing_transport import SigningTransport from fora_sdk.window import Window + from ._call import Answer + EXCHANGE_SERVICE = "fora.v1.ExchangeService" BROKER_SERVICE = "fora.v1.BrokerService" CATALOG_SERVICE = "fora.v1.CatalogService" @@ -157,9 +159,9 @@ class ClientConfig: #: envelope may carry only ``code``, ``message`` and ``details``, must name a known #: Connect code and carry well-formed details, and every ErrorDetail in it, its binary #: ``value`` and its ``debug`` projection alike, must pass the strict ErrorDetail - #: schema and the cross-field rules. A refusal is MALFORMED with ``code`` and - #: ``status`` kept and no detail. Off by default, because the models accept fields a - #: newer minor version may add. + #: schema and the cross-field rules. A refusal is MALFORMED with ``code``, ``status`` + #: and ``headers`` kept and no detail. Off by default, because the models accept fields + #: a newer minor version may add. strict: bool = False #: The freshness window stamped on a delivery-fetch proof. proof_window: Window | None = None @@ -167,6 +169,19 @@ class ClientConfig: call_timeout_sec: float = DEFAULT_CALL_TIMEOUT_SEC content_timeout_sec: float | None = None max_content_bytes: int | None = None + #: Refuse a request the contract does not describe, before it is signed. A verb then + #: takes only its own generated request model from ``wire.models``, or a + #: :class:`RawBody`, which is sent exactly as given and never checked: a plain dict, + #: or a model of another message, is refused as MALFORMED before anything is + #: resolved, read or sent. The message as stamped, with ``ver``, the configured + #: ``requester`` and a purchase's reflected offers in it, must then pass the + #: published strict JSON Schema of the request message and the SDK's cross-field + #: rules, so an unknown member at any depth is refused and named. Applies whatever + #: ``validation`` says. Off by default, because a dict is the common way to build a + #: request and a purchase reflects offers that may carry a member a newer minor + #: version added. A keyword argument a generated model does not declare is dropped + #: by the model before the SDK sees it, so that mistake is a type checker's to catch. + strict_requests: bool = False def resolved_verifier(self) -> Verifier: if self.verifier is not None: @@ -241,6 +256,19 @@ class Plan: strict: bool = False +def decode_answer(plan: Plan, answer: Answer, model: type[BaseModel]) -> Any: + """Decode one answer into ``model`` as the plan says, handing every failure built from + it the answer's header fields.""" + return decode( + plan.op, + answer.status, + answer.body, + model, + strict=plan.strict, + headers=answer.headers, + ) + + # --------------------------------------------------------------------------- # discover # --------------------------------------------------------------------------- @@ -264,13 +292,22 @@ def plan_discover(cfg: ClientConfig, query: RequestMessage) -> Plan: route = _Route(op, cfg.base_url, EXCHANGE_SERVICE, "DiscoverResources") if isinstance(query, RawBody): return _plan(cfg, route, query) - sent = _stamp_discovery(op, query, cfg.requester) - validate_request(op, sent, ResourceQuery, cfg.validation) + sent = _stamp_discovery( + op, query, cfg.requester, model=ResourceQuery, strict_requests=cfg.strict_requests + ) + validate_request(op, sent, ResourceQuery, cfg.validation, strict_requests=cfg.strict_requests) return _plan(cfg, route, sent) -def finish_discover(cfg: ClientConfig, plan: Plan, status: int, body: str) -> DiscoveryResult: - msg, raw = decode_with_raw(plan.op, status, body, ResourceResponse, strict=plan.strict) +def finish_discover(cfg: ClientConfig, plan: Plan, answer: Answer) -> DiscoveryResult: + msg, raw = decode_with_raw( + plan.op, + answer.status, + answer.body, + ResourceResponse, + strict=plan.strict, + headers=answer.headers, + ) return DiscoveryResult( # The offers are read from the RAW answer, not the parsed one. A model parse is # the GATE — it proves the answer well formed and its field names canonical — but @@ -370,18 +407,22 @@ def plan_resolve(cfg: ClientConfig, request: RequestMessage) -> Plan: op = "resolve" if isinstance(request, RawBody): return _plan(cfg, _Route(op, cfg.base_url, BROKER_SERVICE, "Resolve"), request) - sent = _stamp_discovery(op, request, cfg.requester) + sent = _stamp_discovery( + op, request, cfg.requester, model=DiscoveryRequest, strict_requests=cfg.strict_requests + ) # Refused locally rather than sent: a Broker resolves the calling agent from the # requester and declines a request that names none, so this is a verdict the client # already knows, and naming the remedy beats relaying "requester required" from a # round trip away. execute refuses the same way. if sent.get("requester") is None: raise malformed(op, "no requester configured; a Broker resolves who is asking") - validate_request(op, sent, DiscoveryRequest, cfg.validation) + validate_request( + op, sent, DiscoveryRequest, cfg.validation, strict_requests=cfg.strict_requests + ) return _plan(cfg, _Route(op, cfg.base_url, BROKER_SERVICE, "Resolve"), sent) -def finish_resolve(cfg: ClientConfig, plan: Plan, status: int, body: str) -> DiscoveryResult: +def finish_resolve(cfg: ClientConfig, plan: Plan, answer: Answer) -> DiscoveryResult: """Read the Broker's answer. Every returned offer is verified through the SAME fail-closed Verifier discover uses — @@ -392,7 +433,14 @@ def finish_resolve(cfg: ClientConfig, plan: Plan, status: int, body: str) -> Dis A resolve that finds nothing is a SUCCESSFUL answer carrying a typed reason, not a failure. """ - msg, raw = decode_with_raw(plan.op, status, body, DiscoveryResponse, strict=plan.strict) + msg, raw = decode_with_raw( + plan.op, + answer.status, + answer.body, + DiscoveryResponse, + strict=plan.strict, + headers=answer.headers, + ) groups = raw.get("offer_groups") return DiscoveryResult( groups=cfg.resolved_verifier().sort_groups( @@ -432,7 +480,9 @@ def plan_execute( offers = [offer] if isinstance(offer, VerifiedOffer) else list(offer) _require_one_exchange(op, offers) sent = _build_transaction(cfg, op, offers, idempotency_key) - validate_request(op, sent, TransactionRequest, cfg.validation) + validate_request( + op, sent, TransactionRequest, cfg.validation, strict_requests=cfg.strict_requests + ) return _plan(cfg, _Route(op, cfg.base_url, EXCHANGE_SERVICE, "ExecuteTransaction"), sent) @@ -469,11 +519,13 @@ def plan_broker_execute( if cfg.signer is not None: _require_requester_is_signer(op, cfg.requester, cfg.signer.signature_agent) sent = _build_transaction(cfg, op, offers, idempotency_key) - validate_request(op, sent, TransactionRequest, cfg.validation) + validate_request( + op, sent, TransactionRequest, cfg.validation, strict_requests=cfg.strict_requests + ) return _plan(cfg, _Route(op, cfg.base_url, BROKER_SERVICE, "ExecuteTransaction"), sent) -def finish_broker_execute(plan: Plan, status: int, body: str) -> BrokerTransactionResponse: +def finish_broker_execute(plan: Plan, answer: Answer) -> BrokerTransactionResponse: """Read the Broker's combined answer. An Exchange that refused the Broker's whole sub-request is NOT a failure here: the call @@ -485,9 +537,7 @@ def finish_broker_execute(plan: Plan, status: int, body: str) -> BrokerTransacti or directly: each Exchange scopes the key to the agent whatever the route. Only the Broker's own refusals raise. """ - return decode( # type: ignore[no-any-return] - plan.op, status, body, BrokerTransactionResponse, strict=plan.strict - ) + return decode_answer(plan, answer, BrokerTransactionResponse) # type: ignore[no-any-return] def _offer_wire(offer: VerifiedOffer) -> dict[str, Any]: @@ -657,11 +707,9 @@ def _require_requester_is_signer(op: str, requester: dict[str, Any], signature_a ) -def finish_execute(plan: Plan, status: int, body: str) -> TransactionResponse: +def finish_execute(plan: Plan, answer: Answer) -> TransactionResponse: """Read the Exchange's answer.""" - return decode( # type: ignore[no-any-return] - plan.op, status, body, TransactionResponse, strict=plan.strict - ) + return decode_answer(plan, answer, TransactionResponse) # type: ignore[no-any-return] # --------------------------------------------------------------------------- @@ -694,8 +742,8 @@ def plan_report_usage( ) -def finish_report_usage(plan: Plan, status: int, body: str) -> UsageReportResponse: - return decode(plan.op, status, body, UsageReportResponse, strict=plan.strict) # type: ignore[no-any-return] +def finish_report_usage(plan: Plan, answer: Answer) -> UsageReportResponse: + return decode_answer(plan, answer, UsageReportResponse) # type: ignore[no-any-return] def plan_dispute(cfg: ClientConfig, request: RequestMessage, idempotency_key: str | None) -> Plan: @@ -714,8 +762,8 @@ def plan_dispute(cfg: ClientConfig, request: RequestMessage, idempotency_key: st ) -def finish_dispute(plan: Plan, status: int, body: str) -> DisputeResponse: - return decode(plan.op, status, body, DisputeResponse, strict=plan.strict) # type: ignore[no-any-return] +def finish_dispute(plan: Plan, answer: Answer) -> DisputeResponse: + return decode_answer(plan, answer, DisputeResponse) # type: ignore[no-any-return] @dataclass(frozen=True) @@ -737,7 +785,13 @@ def _plan_offer_derived( # operator's own configuration. The same split the Go client makes. if isinstance(message, RawBody): return _plan_raw_routed(cfg, verb.op, verb.method, message) - sent = _stamp_envelope(verb.op, message, idempotency_key) + sent = _stamp_envelope( + verb.op, + message, + idempotency_key, + model=verb.model, + strict_requests=cfg.strict_requests, + ) return _plan_routed_keyless(cfg, _Routed(verb.op, verb.model, verb.method), sent) @@ -782,7 +836,7 @@ def plan_register(cfg: ClientConfig, request: RequestMessage) -> Plan: op = "register" if isinstance(request, RawBody): return _plan_raw_routed(cfg, op, "Register", request) - sent = _stamp_ver(op, request) + sent = _stamp_ver(op, request, model=RegisterRequest, strict_requests=cfg.strict_requests) _require_recipient(op, _str_field(sent, "exchange")) data = sent.get("registration_data") verdict = check_registration_data(data if isinstance(data, dict) else None) @@ -793,8 +847,8 @@ def plan_register(cfg: ClientConfig, request: RequestMessage) -> Plan: return _plan_routed_keyless(cfg, _Routed(op, RegisterRequest, "Register"), sent) -def finish_register(plan: Plan, status: int, body: str) -> RegisterResponse: - return decode(plan.op, status, body, RegisterResponse, strict=plan.strict) # type: ignore[no-any-return] +def finish_register(plan: Plan, answer: Answer) -> RegisterResponse: + return decode_answer(plan, answer, RegisterResponse) # type: ignore[no-any-return] def plan_get_account_status(cfg: ClientConfig, request: RequestMessage) -> Plan: @@ -812,13 +866,15 @@ def plan_get_account_status(cfg: ClientConfig, request: RequestMessage) -> Plan: op = "get account status" if isinstance(request, RawBody): return _plan_raw_routed(cfg, op, "GetAccountStatus", request) - sent = _stamp_ver(op, request) + sent = _stamp_ver( + op, request, model=GetAccountStatusRequest, strict_requests=cfg.strict_requests + ) _require_recipient(op, _str_field(sent, "exchange")) return _plan_routed_keyless(cfg, _Routed(op, GetAccountStatusRequest, "GetAccountStatus"), sent) -def finish_get_account_status(plan: Plan, status: int, body: str) -> GetAccountStatusResponse: - return decode(plan.op, status, body, GetAccountStatusResponse, strict=plan.strict) # type: ignore[no-any-return] +def finish_get_account_status(plan: Plan, answer: Answer) -> GetAccountStatusResponse: + return decode_answer(plan, answer, GetAccountStatusResponse) # type: ignore[no-any-return] def _apply_registration_requirements(cfg: ClientConfig, op: str, sent: dict[str, Any]) -> None: @@ -912,7 +968,7 @@ def _plan_routed_keyless(cfg: ClientConfig, verb: _Routed, sent: dict[str, Any]) """ op = verb.op endpoint = vet_exchange_endpoint(cfg.endpoint_resolver, _str_field(sent, "exchange"), op) - validate_request(op, sent, verb.model, cfg.validation) + validate_request(op, sent, verb.model, cfg.validation, strict_requests=cfg.strict_requests) # The manifest-derived leg, so the guarded transport: the caller named a domain, the # manifest it serves named this endpoint, and a signed call now goes there. return _plan(cfg, _Route(op, endpoint, EXCHANGE_SERVICE, verb.method), sent, guarded=True) @@ -942,8 +998,8 @@ def plan_push_resources(cfg: ClientConfig, request: RequestMessage) -> Plan: return _plan_catalog(cfg, _PUSH_RESOURCES, request) -def finish_push_resources(plan: Plan, status: int, body: str) -> PushResourcesResponse: - return decode(plan.op, status, body, PushResourcesResponse, strict=plan.strict) +def finish_push_resources(plan: Plan, answer: Answer) -> PushResourcesResponse: + return decode_answer(plan, answer, PushResourcesResponse) def plan_remove_resources(cfg: ClientConfig, request: RequestMessage) -> Plan: @@ -951,8 +1007,8 @@ def plan_remove_resources(cfg: ClientConfig, request: RequestMessage) -> Plan: return _plan_catalog(cfg, _REMOVE_RESOURCES, request) -def finish_remove_resources(plan: Plan, status: int, body: str) -> RemoveResourcesResponse: - return decode(plan.op, status, body, RemoveResourcesResponse, strict=plan.strict) +def finish_remove_resources(plan: Plan, answer: Answer) -> RemoveResourcesResponse: + return decode_answer(plan, answer, RemoveResourcesResponse) def plan_refresh_catalog(cfg: ClientConfig, request: RequestMessage) -> Plan: @@ -960,8 +1016,8 @@ def plan_refresh_catalog(cfg: ClientConfig, request: RequestMessage) -> Plan: return _plan_catalog(cfg, _REFRESH_CATALOG, request) -def finish_refresh_catalog(plan: Plan, status: int, body: str) -> RefreshCatalogResponse: - return decode(plan.op, status, body, RefreshCatalogResponse, strict=plan.strict) +def finish_refresh_catalog(plan: Plan, answer: Answer) -> RefreshCatalogResponse: + return decode_answer(plan, answer, RefreshCatalogResponse) def _plan_catalog(cfg: ClientConfig, verb: _CatalogVerb, message: RequestMessage) -> Plan: @@ -982,14 +1038,20 @@ def _plan_catalog(cfg: ClientConfig, verb: _CatalogVerb, message: RequestMessage route = _Route(op, cfg.base_url, CATALOG_SERVICE, verb.method) if isinstance(message, RawBody): return _plan(cfg, route, message) - sent = _stamp_ver(op, message) + sent = _stamp_ver(op, message, model=verb.model, strict_requests=cfg.strict_requests) _require_recipient(op, _str_field(sent, "exchange")) - validate_request(op, sent, verb.model, cfg.validation) + validate_request(op, sent, verb.model, cfg.validation, strict_requests=cfg.strict_requests) return _plan(cfg, route, sent) -def _stamp_ver(op: str, message: dict[str, Any] | BaseModel) -> dict[str, Any]: - sent = _clone(op, message) +def _stamp_ver( + op: str, + message: dict[str, Any] | BaseModel, + *, + model: type[BaseModel], + strict_requests: bool, +) -> dict[str, Any]: + sent = _clone(op, message, model=model, strict_requests=strict_requests) if _str_field(sent, "ver") == "": sent["ver"] = ProtocolVersion return sent @@ -1029,7 +1091,12 @@ def _require_recipient(op: str, exchange: str) -> None: def _stamp_discovery( - op: str, message: dict[str, Any] | BaseModel, requester: dict[str, Any] | None + op: str, + message: dict[str, Any] | BaseModel, + requester: dict[str, Any] | None, + *, + model: type[BaseModel], + strict_requests: bool, ) -> dict[str, Any]: """Fill the envelope a DISCOVERY call carries, which is the mutating envelope minus the idempotency key: pure discovery buys nothing and changes nothing, so there is no @@ -1040,7 +1107,7 @@ def _stamp_discovery( requester is filled because a query must name one, while the client already holds that identity. """ - sent = _clone(op, message) + sent = _clone(op, message, model=model, strict_requests=strict_requests) if not sent.get("ver"): sent["ver"] = ProtocolVersion if sent.get("requester") is None and requester is not None: @@ -1049,7 +1116,12 @@ def _stamp_discovery( def _stamp_envelope( - op: str, message: dict[str, Any] | BaseModel, idempotency_key: str | None + op: str, + message: dict[str, Any] | BaseModel, + idempotency_key: str | None, + *, + model: type[BaseModel], + strict_requests: bool, ) -> dict[str, Any]: """Fill the two envelope fields the protocol requires on a state-mutating call, WITHOUT overwriting what the caller already set. @@ -1060,7 +1132,7 @@ def _stamp_envelope( there is theirs — discarding it would turn each of their retries into a fresh action, which is the double-counting the field exists to prevent. A pinned key overrides both. """ - sent = _clone(op, message) + sent = _clone(op, message, model=model, strict_requests=strict_requests) if not sent.get("ver"): sent["ver"] = ProtocolVersion on_message = sent.get("idempotency_key") @@ -1093,14 +1165,33 @@ def _rate_limit(parsed: Any, wire: Any) -> dict[str, Any] | None: return standing -def _clone(op: str, message: dict[str, Any] | BaseModel) -> dict[str, Any]: +def _clone( + op: str, + message: dict[str, Any] | BaseModel, + *, + model: type[BaseModel], + strict_requests: bool, +) -> dict[str, Any]: """Copy a caller's message so the SDK can stamp its envelope without touching what the caller still holds. A deep copy, because the envelope fields are top-level but a caller re-using a nested object across calls must not see it change either. A generated model is rendered through :func:`to_wire` instead, which already returns a fresh object. This is the one point every dict-taking verb passes through, so a - model and the equivalent dict are serialized the same way exactly once.""" + model and the equivalent dict are serialized the same way exactly once — and so it is + where ``strict_requests`` refuses anything but the verb's own ``model``, before the + verb resolves, reads or sends anything.""" + if strict_requests and not isinstance(message, model): + given = ( + "a plain dict" + if isinstance(message, dict) + else f"a value of type {type(message).__name__}" + ) + raise malformed( + op, + f"strict_requests refuses {given}; pass wire.models.{model.__name__}, " + "or RawBody to send a body exactly as given", + ) if isinstance(message, BaseModel): return to_wire(message) try: diff --git a/sdk/python/fora_sdk/client/content.py b/sdk/python/fora_sdk/client/content.py index c769bb93..0a41f431 100644 --- a/sdk/python/fora_sdk/client/content.py +++ b/sdk/python/fora_sdk/client/content.py @@ -284,6 +284,7 @@ def edge_refusal(response: httpx.Response, body: bytes) -> CallError: status=response.status_code, reason=token or None, detail=_typed_reason(token) if token else None, + headers=response.headers, ) diff --git a/sdk/python/fora_sdk/client/errors.py b/sdk/python/fora_sdk/client/errors.py index 53c1fea1..bdfd043d 100644 --- a/sdk/python/fora_sdk/client/errors.py +++ b/sdk/python/fora_sdk/client/errors.py @@ -14,7 +14,11 @@ import enum from typing import TYPE_CHECKING +import httpx + if TYPE_CHECKING: + from collections.abc import Mapping + from wire.models import ErrorDetail #: The reason a peer's answer is refused for spelling its field names the wrong way. @@ -26,6 +30,9 @@ #: clients can observe, not a divergence in the shared failure taxonomy. NOT_CANONICAL_WIRE_NAMING = "not_canonical_wire_naming" +#: The status from which an answer is an error, and its header fields a refusal's. +_ERROR_STATUS = 400 + class CallErrorKind(enum.Enum): """Why a client call did not produce an answer.""" @@ -70,6 +77,7 @@ def __init__( peer_message: str = "", cause: BaseException | str | None = None, code: str | None = None, + headers: Mapping[str, str] | None = None, ) -> None: self.kind = kind #: The verb that failed, in the SDK's own words ("discover", "fetch content"). @@ -110,6 +118,20 @@ def __init__( #: refusals are edge tokens rather than Connect codes. ``reason`` may hold either #: kind of token; this field holds only the Connect code. self.code = code + #: The header fields of the peer's error answer, 400 or above: an + #: ``Accept-Signature`` naming what a verifier requires, a ``Retry-After``, an + #: edge's discovery headers, which ``parse_discovery_hint(err.status or 0, + #: err.headers)`` reads. Empty for any other failure — a local one, a transport one, a + #: 2xx this client could not read, and a refused redirect, whose ``Location`` + #: names a URL the peer chose and so must not reach a caller here. + #: + #: A copy, read case-insensitively; a field sent on several lines reads as one + #: value joined with ", ". The values are the peer's, unvalidated, and never + #: rendered into the message. + #: + #: The rule is applied here, once: a call site passes the answer's fields and + #: this constructor keeps them only for an error status. + self.headers: Mapping[str, str] = _refusal_headers(status, headers) self.cause = cause super().__init__(_render(kind, op, status, reason, cause)) @@ -119,6 +141,16 @@ def reason_of(self) -> str: return self.reason or self.kind.value +def _refusal_headers( + status: int | None, headers: Mapping[str, str] | None +) -> Mapping[str, str]: + """The fields a failure carries for the answer that produced it: a copy for an error + status, nothing for any other answer.""" + if headers is None or status is None or status < _ERROR_STATUS: + return httpx.Headers() + return httpx.Headers(headers) + + def _render( kind: CallErrorKind, op: str, diff --git a/sdk/python/fora_sdk/core.py b/sdk/python/fora_sdk/core.py index df1c6074..f4c387c4 100644 --- a/sdk/python/fora_sdk/core.py +++ b/sdk/python/fora_sdk/core.py @@ -33,6 +33,7 @@ import base64 import enum import re +from collections.abc import Callable from dataclasses import dataclass, field from datetime import UTC, datetime from typing import TYPE_CHECKING, Any @@ -40,7 +41,7 @@ import rfc8785 if TYPE_CHECKING: - from collections.abc import Callable, Sequence + from collections.abc import Sequence from cryptography.exceptions import InvalidSignature from cryptography.hazmat.primitives.asymmetric.ed25519 import ( Ed25519PrivateKey, @@ -450,9 +451,35 @@ def jcs_acceptance_payload( concrete offer. Fail-closed the same way on an empty ``requester_id`` or ``requester_domain``: an acceptance must name the requester it binds. """ + return _acceptance_payload( + offer_sig=offer_sig, + requester_id=requester_id, + requester_domain=requester_domain, + idempotency_key=idempotency_key, + requester_check=_require_named_requester, + ) + + +#: Refuses a requester an acceptance may not name, given its id and domain, by raising +#: ``ValueError``. The protocol's signer passes :func:`_require_named_requester`; +#: :mod:`fora_sdk.testing` passes one that admits an empty requester. +_RequesterCheck = Callable[[str, str], None] + + +def _acceptance_payload( + *, + offer_sig: str, + requester_id: str, + requester_domain: str, + idempotency_key: str, + requester_check: _RequesterCheck, +) -> bytes: + """The canonical AgentAcceptancePayload bytes, refusing an unsigned offer and then + whatever ``requester_check`` refuses. The one renderer both the protocol's signer and + the test signers use.""" if offer_sig == "": raise ValueError("cannot accept an unsigned offer (empty offer signature)") - _require_named_requester(requester_id, requester_domain) + requester_check(requester_id, requester_domain) payload: dict[str, str] = { "offer_sig": offer_sig, "requester_id": requester_id, @@ -462,6 +489,12 @@ def jcs_acceptance_payload( return rfc8785.dumps({k: v for k, v in payload.items() if v != ""}) +def _sign_payload(seed: bytes, payload: bytes) -> tuple[str, str]: + """Sign canonical acceptance bytes; return ``(hex_signature, "EdDSA")``.""" + priv = Ed25519PrivateKey.from_private_bytes(seed) + return priv.sign(payload).hex(), ACCEPTANCE_SIGNATURE_ALGORITHM + + #: The shape a hex-encoded signature may have. Checked BEFORE ``bytes.fromhex``, which #: skips ASCII whitespace BETWEEN bytes — CPython has done so since 3.7 — and would read #: ``"00 ff"`` as two bytes where Go's ``hex.DecodeString`` and the TypeScript decoder both @@ -496,8 +529,7 @@ def sign_offer_acceptance_jcs( requester_domain=requester_domain, idempotency_key=idempotency_key, ) - priv = Ed25519PrivateKey.from_private_bytes(seed) - return priv.sign(payload).hex(), ACCEPTANCE_SIGNATURE_ALGORITHM + return _sign_payload(seed, payload) def verify_offer_acceptance_jcs( @@ -543,7 +575,27 @@ def jcs_request_acceptance_payload( Raises ``ValueError`` on no items, an item with an empty offer signature or exchange, and an empty ``requester_id`` or ``requester_domain`` (see the offer acceptance). """ - _require_named_requester(requester_id, requester_domain) + return _request_acceptance_payload( + items=items, + requester_id=requester_id, + requester_domain=requester_domain, + idempotency_key=idempotency_key, + requester_check=_require_named_requester, + ) + + +def _request_acceptance_payload( + *, + items: list[tuple[str, str]], + requester_id: str, + requester_domain: str, + idempotency_key: str, + requester_check: _RequesterCheck, +) -> bytes: + """The canonical request-set bytes, refusing whatever ``requester_check`` refuses, then + no items, then an item's empty offer signature or exchange. The one renderer both the + protocol's signer and the test signers use.""" + requester_check(requester_id, requester_domain) if not items: raise ValueError("request acceptance requires at least one item") refs: list[dict[str, str]] = [] @@ -579,8 +631,7 @@ def sign_request_acceptance_jcs( requester_domain=requester_domain, idempotency_key=idempotency_key, ) - priv = Ed25519PrivateKey.from_private_bytes(seed) - return priv.sign(payload).hex(), ACCEPTANCE_SIGNATURE_ALGORITHM + return _sign_payload(seed, payload) def verify_request_acceptance_jcs( diff --git a/sdk/python/fora_sdk/sync/__init__.py b/sdk/python/fora_sdk/sync/__init__.py index c9017a62..162b577a 100644 --- a/sdk/python/fora_sdk/sync/__init__.py +++ b/sdk/python/fora_sdk/sync/__init__.py @@ -18,7 +18,7 @@ import httpx from fora_sdk.client import _admin, _fetch_inputs, _verbs -from fora_sdk.client._call import as_call_error +from fora_sdk.client._call import Answer, as_call_error from fora_sdk.client._read import ( IDENTITY_ENCODING, bounded_chunks, @@ -121,7 +121,7 @@ def _refuse_if_closed(self, op: str) -> None: CallErrorKind.NOT_SENT, op, cause="the client has been closed" ) - def _send(self, plan: _verbs.Plan) -> tuple[int, str]: + def _send(self, plan: _verbs.Plan) -> Answer: """Send one RPC and read its answer under the cap. The async twin's own docstring carries the reasoning; only the iteration differs.""" self._refuse_if_closed(plan.op) @@ -140,10 +140,10 @@ def _send(self, plan: _verbs.Plan) -> tuple[int, str]: follow_redirects=False, ) as response: refuse_unrequested_encoding(plan.op, response) - read = bounded_chunks(plan.op, plan.max_bytes, response.status_code) + read = bounded_chunks(plan.op, plan.max_bytes, response) for chunk in response.iter_bytes(): read.add(chunk) - return response.status_code, read.text() + return Answer(response.status_code, read.text(), response.headers) except httpx.HTTPError as exc: raise as_call_error(plan.op, exc) from exc except _ssrf.SsrfError as exc: @@ -165,8 +165,8 @@ def __init__(self, config: ClientConfig, *, http: httpx.Client | None = None) -> def discover(self, query: RequestMessage) -> DiscoveryResult: plan = _verbs.plan_discover(self._config, query) - status, body = self._send(plan) - return _verbs.finish_discover(self._config, plan, status, body) + answer = self._send(plan) + return _verbs.finish_discover(self._config, plan, answer) def execute( self, @@ -175,32 +175,32 @@ def execute( idempotency_key: str | None = None, ) -> TransactionResponse: plan = _verbs.plan_execute(self._config, offer, idempotency_key) - status, body = self._send(plan) - return _verbs.finish_execute(plan, status, body) + answer = self._send(plan) + return _verbs.finish_execute(plan, answer) def report_usage( self, report: RequestMessage, *, idempotency_key: str | None = None ) -> UsageReportResponse: plan = _verbs.plan_report_usage(self._config, report, idempotency_key) - status, body = self._send(plan) - return _verbs.finish_report_usage(plan, status, body) + answer = self._send(plan) + return _verbs.finish_report_usage(plan, answer) def dispute( self, request: RequestMessage, *, idempotency_key: str | None = None ) -> DisputeResponse: plan = _verbs.plan_dispute(self._config, request, idempotency_key) - status, body = self._send(plan) - return _verbs.finish_dispute(plan, status, body) + answer = self._send(plan) + return _verbs.finish_dispute(plan, answer) def register(self, request: RequestMessage) -> RegisterResponse: plan = _verbs.plan_register(self._config, request) - status, body = self._send(plan) - return _verbs.finish_register(plan, status, body) + answer = self._send(plan) + return _verbs.finish_register(plan, answer) def get_account_status(self, request: RequestMessage) -> GetAccountStatusResponse: plan = _verbs.plan_get_account_status(self._config, request) - status, body = self._send(plan) - return _verbs.finish_get_account_status(plan, status, body) + answer = self._send(plan) + return _verbs.finish_get_account_status(plan, answer) def fetch(self, signed_url: str) -> Content: url = signed_url @@ -231,6 +231,7 @@ def fetch(self, signed_url: str) -> Content: op, status=response.status_code, cause="peer answered with a redirect, which this client does not follow", + headers=response.headers, ) if not response.is_success: refusal = b"" @@ -239,7 +240,7 @@ def fetch(self, signed_url: str) -> Content: if len(refusal) >= MAX_ERROR_BODY_BYTES: break raise edge_refusal(response, refusal[:MAX_ERROR_BODY_BYTES]) - read = bounded_chunks(op, max_bytes, response.status_code) + read = bounded_chunks(op, max_bytes, response) for chunk in response.iter_bytes(): read.add(chunk) return read_content(url, response, read.body()) @@ -259,15 +260,15 @@ def __init__(self, config: ClientConfig, *, http: httpx.Client | None = None) -> def resolve(self, request: RequestMessage) -> DiscoveryResult: plan = _verbs.plan_resolve(self._config, request) - status, body = self._send(plan) - return _verbs.finish_resolve(self._config, plan, status, body) + answer = self._send(plan) + return _verbs.finish_resolve(self._config, plan, answer) def execute( self, offers: Sequence[VerifiedOffer] | RawBody, *, idempotency_key: str | None = None ) -> BrokerTransactionResponse: plan = _verbs.plan_broker_execute(self._config, offers, idempotency_key) - status, body = self._send(plan) - return _verbs.finish_broker_execute(plan, status, body) + answer = self._send(plan) + return _verbs.finish_broker_execute(plan, answer) class CatalogClient(_Face): @@ -278,18 +279,18 @@ def __init__(self, config: ClientConfig, *, http: httpx.Client | None = None) -> def push_resources(self, request: RequestMessage) -> PushResourcesResponse: plan = _verbs.plan_push_resources(self._config, request) - status, body = self._send(plan) - return _verbs.finish_push_resources(plan, status, body) + answer = self._send(plan) + return _verbs.finish_push_resources(plan, answer) def remove_resources(self, request: RequestMessage) -> RemoveResourcesResponse: plan = _verbs.plan_remove_resources(self._config, request) - status, body = self._send(plan) - return _verbs.finish_remove_resources(plan, status, body) + answer = self._send(plan) + return _verbs.finish_remove_resources(plan, answer) def refresh_catalog(self, request: RequestMessage) -> RefreshCatalogResponse: plan = _verbs.plan_refresh_catalog(self._config, request) - status, body = self._send(plan) - return _verbs.finish_refresh_catalog(plan, status, body) + answer = self._send(plan) + return _verbs.finish_refresh_catalog(plan, answer) class AdminClient(_Face): @@ -300,20 +301,20 @@ def __init__(self, config: ClientConfig, *, http: httpx.Client | None = None) -> def set_tenant_fee_rate(self, request: RequestMessage) -> SetTenantFeeRateResponse: plan = _admin.plan_set_tenant_fee_rate(self._config, request) - status, body = self._send(plan) - return _admin.finish_set_tenant_fee_rate(plan, status, body) + answer = self._send(plan) + return _admin.finish_set_tenant_fee_rate(plan, answer) def set_reporting_policy(self, request: RequestMessage) -> SetReportingPolicyResponse: plan = _admin.plan_set_reporting_policy(self._config, request) - status, body = self._send(plan) - return _admin.finish_set_reporting_policy(plan, status, body) + answer = self._send(plan) + return _admin.finish_set_reporting_policy(plan, answer) def request_domain_verification(self, request: RequestMessage) -> DomainVerificationChallenge: plan = _admin.plan_request_domain_verification(self._config, request) - status, body = self._send(plan) - return _admin.finish_request_domain_verification(plan, status, body) + answer = self._send(plan) + return _admin.finish_request_domain_verification(plan, answer) def confirm_domain_verification(self, request: RequestMessage) -> DomainVerificationResult: plan = _admin.plan_confirm_domain_verification(self._config, request) - status, body = self._send(plan) - return _admin.finish_confirm_domain_verification(plan, status, body) + answer = self._send(plan) + return _admin.finish_confirm_domain_verification(plan, answer) diff --git a/sdk/python/fora_sdk/testing.py b/sdk/python/fora_sdk/testing.py new file mode 100644 index 00000000..0ea17786 --- /dev/null +++ b/sdk/python/fora_sdk/testing.py @@ -0,0 +1,107 @@ +"""Acceptance signers that admit an empty requester, for a test proving a verifier refuses +them. + +fora.proto requires an acceptance to name its requester: "A signer refuses to sign bytes +that name an empty requester, and a verifier refuses an acceptance whose canonical bytes +name one, even when the signature over those bytes verifies." The signers in +:mod:`fora_sdk.core` are that signer, so they cannot produce the one input a verifier's +refusal is tested with. The four functions here are their twins with that single refusal +skipped. They render through the same canonical form and keep every other refusal, so an +acceptance they sign is valid in every respect but the requester, and a verifier's refusal +of it is attributable to that alone. The ``refused`` lists of the shared +``acceptance-vectors.json`` and ``request-acceptance-vectors.json`` record exactly what they +produce. + +This module is test support: nothing in the SDK imports it, it is not re-exported from +:mod:`fora_sdk`, and an acceptance it signs is one a conforming party must never send. Go's +``helperstest`` and TypeScript's ``@fora-protocol/sdk/testing`` are its twins. +""" + +from __future__ import annotations + +from fora_sdk.core import _acceptance_payload, _request_acceptance_payload, _sign_payload + +__all__ = [ + "jcs_acceptance_payload_any_requester", + "jcs_request_acceptance_payload_any_requester", + "sign_offer_acceptance_any_requester", + "sign_request_acceptance_any_requester", +] + + +def _any_requester(_requester_id: str, _requester_domain: str) -> None: + """Admit every requester, an empty one included.""" + + +def jcs_acceptance_payload_any_requester( + *, + offer_sig: str, + requester_id: str, + requester_domain: str, + idempotency_key: str, +) -> bytes: + """:func:`fora_sdk.core.jcs_acceptance_payload` without its refusal of an empty + requester id or domain.""" + return _acceptance_payload( + offer_sig=offer_sig, + requester_id=requester_id, + requester_domain=requester_domain, + idempotency_key=idempotency_key, + requester_check=_any_requester, + ) + + +def sign_offer_acceptance_any_requester( + *, + seed: bytes, + offer_sig: str, + requester_id: str, + requester_domain: str, + idempotency_key: str, +) -> tuple[str, str]: + """:func:`fora_sdk.core.sign_offer_acceptance_jcs` without its refusal of an empty + requester id or domain; returns ``(hex_signature, "EdDSA")``.""" + payload = jcs_acceptance_payload_any_requester( + offer_sig=offer_sig, + requester_id=requester_id, + requester_domain=requester_domain, + idempotency_key=idempotency_key, + ) + return _sign_payload(seed, payload) + + +def jcs_request_acceptance_payload_any_requester( + *, + items: list[tuple[str, str]], + requester_id: str, + requester_domain: str, + idempotency_key: str, +) -> bytes: + """:func:`fora_sdk.core.jcs_request_acceptance_payload` without its refusal of an + empty requester id or domain.""" + return _request_acceptance_payload( + items=items, + requester_id=requester_id, + requester_domain=requester_domain, + idempotency_key=idempotency_key, + requester_check=_any_requester, + ) + + +def sign_request_acceptance_any_requester( + *, + seed: bytes, + items: list[tuple[str, str]], + requester_id: str, + requester_domain: str, + idempotency_key: str, +) -> tuple[str, str]: + """:func:`fora_sdk.core.sign_request_acceptance_jcs` without its refusal of an empty + requester id or domain; returns ``(hex_signature, "EdDSA")``.""" + payload = jcs_request_acceptance_payload_any_requester( + items=items, + requester_id=requester_id, + requester_domain=requester_domain, + idempotency_key=idempotency_key, + ) + return _sign_payload(seed, payload) diff --git a/sdk/python/fora_sdk/wire_canon.py b/sdk/python/fora_sdk/wire_canon.py index 5241f853..4a27b3c5 100644 --- a/sdk/python/fora_sdk/wire_canon.py +++ b/sdk/python/fora_sdk/wire_canon.py @@ -5,8 +5,8 @@ That is what ``sdk/go/connectserver/codec.go`` marshals (``UseProtoNames: true``), and what ``codec_test.go`` guards — a stray ``UseProtoNames=false`` is the regression it catches. The offer SIGNATURE covers the CANONICAL form: the same snake_case proto -names, but omit-unpopulated, enums-as-names (sdk/go helpers -``canonicalSignJSONOptions``), then RFC 8785 JCS. Both sides share the naming, so what +names, but omit-unpopulated, enums-as-names (``JSONOptions`` in +sdk/go/internal/canonicalsign), then RFC 8785 JCS. Both sides share the naming, so what this inversion undoes is the zero-inflation, NOT the naming. It does NOT clear the signature fields — :func:`from_wire_offer` keeps ``signature`` / ``signature_algorithm`` verbatim, and diff --git a/sdk/python/pyproject.toml b/sdk/python/pyproject.toml index 4860c8d0..a6fd7b6c 100644 --- a/sdk/python/pyproject.toml +++ b/sdk/python/pyproject.toml @@ -80,6 +80,12 @@ select = [ # second type that exists only to satisfy an argument count. PLR0913 is waived for exactly # this constructor. "fora_sdk/client/errors.py" = ["PLR0913"] +# The decode step reads one answer — the verb, its status, body and header fields — into +# one model under the strict flag. The answer's parts stay separate arguments because the +# shared corpora replay them as separate columns straight into decode, so folding them into +# an object would make every replay assemble one; the client faces pass their Answer +# through decode_answer instead. PLR0913 is waived for exactly this module. +"fora_sdk/client/_call.py" = ["PLR0913"] # The L2 core mirrors the same crypto-primitive shape: the offer-verify _check # walks each fail-closed rejection reason as its own early return (resolver-miss, # bad-hex, sig-invalid, expired), and the acceptance verify keeps the four diff --git a/sdk/python/tests/test_acceptance_refusals.py b/sdk/python/tests/test_acceptance_refusals.py new file mode 100644 index 00000000..4fa22b69 --- /dev/null +++ b/sdk/python/tests/test_acceptance_refusals.py @@ -0,0 +1,91 @@ +"""The refusals of the acceptance canonicalizers and signers, pinned. + +The exact text, and which refusal wins when an input has several faults. The renderer and +the checks around it are shared with code that skips one of them (``fora_sdk.testing``), so +the order and the wording are part of what must not move; downstream suites match the text. +""" + +from __future__ import annotations + +import re +from typing import Any + +import pytest + +from fora_sdk.core import ( + jcs_acceptance_payload, + jcs_request_acceptance_payload, + sign_offer_acceptance_jcs, + sign_request_acceptance_jcs, + verify_offer_acceptance_jcs, + verify_request_acceptance_jcs, +) + +_SEED = bytes(range(32)) +_NAMED = {"requester_id": "agent-1", "requester_domain": "agent.example", "idempotency_key": "idem"} +_UNSIGNED = "cannot accept an unsigned offer (empty offer signature)" +_NO_ID = "acceptance names an empty requester: requester_id is empty" +_NO_DOMAIN = "acceptance names an empty requester: requester_domain is empty" +_NO_ITEMS = "request acceptance requires at least one item" + + +def _with(**changes: Any) -> dict[str, Any]: + return {**_NAMED, **changes} + + +@pytest.mark.parametrize( + ("fields", "message"), + [ + (_with(offer_sig=""), _UNSIGNED), + (_with(offer_sig="sig", requester_id=""), _NO_ID), + (_with(offer_sig="sig", requester_domain=""), _NO_DOMAIN), + # An unsigned offer is refused before an empty requester. + (_with(offer_sig="", requester_id=""), _UNSIGNED), + (_with(offer_sig="sig", requester_id="", requester_domain=""), _NO_ID), + ], +) +def test_the_offer_acceptance_refusals(fields: dict[str, Any], message: str) -> None: + for call in ( + lambda: jcs_acceptance_payload(**fields), + lambda: sign_offer_acceptance_jcs(seed=_SEED, **fields), + lambda: verify_offer_acceptance_jcs(pubkey_b64="AA==", signature_hex="00", **fields), + ): + with pytest.raises(ValueError, match=f"^{re.escape(message)}$"): + call() + + +_ITEM = ("sig", "one.example") + + +@pytest.mark.parametrize( + ("items", "fields", "message"), + [ + ([], _NAMED, _NO_ITEMS), + ([("", "one.example")], _NAMED, "request acceptance item 0 has an empty offer signature"), + ([("sig", "")], _NAMED, "request acceptance item 0 has an empty exchange"), + ( + [_ITEM, ("", "one.example")], + _NAMED, + "request acceptance item 1 has an empty offer signature", + ), + ([_ITEM], _with(requester_id=""), _NO_ID), + ([_ITEM], _with(requester_domain=""), _NO_DOMAIN), + # An empty requester is refused before no items and before an empty item field. + ([], _with(requester_id=""), _NO_ID), + ([("", "")], _with(requester_domain=""), _NO_DOMAIN), + # Within an item, an empty offer signature is refused before an empty exchange. + ([("", "")], _NAMED, "request acceptance item 0 has an empty offer signature"), + ], +) +def test_the_request_acceptance_refusals( + items: list[tuple[str, str]], fields: dict[str, Any], message: str +) -> None: + for call in ( + lambda: jcs_request_acceptance_payload(items=items, **fields), + lambda: sign_request_acceptance_jcs(seed=_SEED, items=items, **fields), + ): + with pytest.raises(ValueError, match=f"^{re.escape(message)}$"): + call() + assert not verify_request_acceptance_jcs( + pubkey_b64="AA==", signature_hex="00", items=items, **fields + ) diff --git a/sdk/python/tests/test_api_surface_parity.py b/sdk/python/tests/test_api_surface_parity.py index e71bbdf1..c9c487b7 100644 --- a/sdk/python/tests/test_api_surface_parity.py +++ b/sdk/python/tests/test_api_surface_parity.py @@ -7,7 +7,7 @@ the symbol side. Go is the oracle. Every exported Go symbol under -``sdk/go/{helpers,resolvers,core,connect,connectserver,profiles/music}`` must be, in +``sdk/go/{helpers,resolvers,core,connect,connectserver,profiles/music,helperstest}`` must be, in ``sdk/parity/symbol-map.json``, EITHER * a ``symbols`` entry — mapped to the public Python and/or TS name (or null where a @@ -66,7 +66,15 @@ _MAP_PATH = _REPO_ROOT / "sdk" / "parity" / "symbol-map.json" _MATRIX = _REPO_ROOT / "docs" / "sdk-parity-matrix.md" -_GO_PACKAGES = ("helpers", "resolvers", "core", "connect", "connectserver", "profiles/music") +_GO_PACKAGES = ( + "helpers", + "resolvers", + "core", + "connect", + "connectserver", + "profiles/music", + "helperstest", +) # RATCHET — this baseline may only ever DECREASE. Each allowlisted symbol is a # deliberate, DOCUMENTED cross-language divergence, in one of two flavors: @@ -223,10 +231,10 @@ def enumerate_go() -> dict[str, str]: #: in one and absent from the other. def _public_modules() -> list[ModuleType]: import fora_sdk - from fora_sdk import client, music, resolvers + from fora_sdk import client, music, resolvers, testing from fora_sdk import sync as blocking - return [fora_sdk, resolvers, client, blocking, music] + return [fora_sdk, resolvers, client, blocking, music, testing] def enumerate_python() -> set[str]: diff --git a/sdk/python/tests/test_call_error_headers.py b/sdk/python/tests/test_call_error_headers.py new file mode 100644 index 00000000..bee1e7dc --- /dev/null +++ b/sdk/python/tests/test_call_error_headers.py @@ -0,0 +1,265 @@ +"""``CallError.headers``: the header fields of a refusal, and only of a refusal. + +A peer's error answer says what it wants next in its header fields: a Web Bot Auth +verifier names the signature it requires in ``Accept-Signature``, a draining gateway sets +``Retry-After``, a delivery edge's 403 carries the discovery headers. ``headers`` carries +them for an answer of status 400 or above, on both legs and both faces. Every other +failure carries none: a local one, a transport one, a 2xx this client could not read, and a +refused redirect, whose ``Location`` names a URL the peer chose. + +The constructor owns the rule, so every case below is the same rule seen from a different +site that builds the failure. +""" + +from __future__ import annotations + +import json +from typing import Any + +import httpx +import pytest +from test_call_error_code import _Answer, _discover_fails +from test_client import _IDS, FACES, Face, _config + +import fora_sdk.sync as sync_client +from fora_sdk import AcceptSignatureHeader, parse_discovery_hint +from fora_sdk.client import CallError, CallErrorKind, Client +from fora_sdk.client._call import decode +from fora_sdk.pop import POP_ACCEPT_SIGNATURE +from fora_sdk.wba import accept_signature +from fora_sdk.wire import ContentRulesHeader, ExchangeHeader +from wire.models import ResourceResponse + +_UNAUTHENTICATED = json.dumps( + {"code": "unauthenticated", "message": "the request is not signed"} +).encode() +_JSON = {"content-type": "application/json"} +#: A value no message carries by accident, so its absence from ``str(err)`` is evidence. +_PROBE = "probe-6c1f0e" + + +def _assert_not_rendered(err: CallError) -> None: + for name, value in err.headers.items(): + assert value not in str(err), f"the message renders {name}: {err}" + + +# --------------------------------------------------------------------------- +# The rule, at the constructor +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("status", [None, 200, 204, 302, 304, 399]) +def test_no_error_status_carries_no_fields(status: int | None) -> None: + err = CallError(CallErrorKind.REFUSED, "op", status=status, headers={"X-Probe": _PROBE}) + + assert dict(err.headers) == {} + + +@pytest.mark.parametrize("status", [400, 401, 403, 429, 503]) +def test_an_error_status_carries_a_copy_of_its_fields(status: int) -> None: + fields = {"X-Probe": _PROBE} + err = CallError(CallErrorKind.REFUSED, "op", status=status, headers=fields) + fields["X-Probe"] = "changed" + + assert err.headers["X-Probe"] == _PROBE + # Read case-insensitively, as HTTP header fields are. + assert err.headers["x-probe"] == _PROBE + + +def test_a_field_sent_on_two_lines_reads_as_one_joined_value() -> None: + lines = httpx.Headers([("X-Probe", f"{_PROBE}-first"), ("X-Probe", f"{_PROBE}-second")]) + err = CallError(CallErrorKind.REFUSED, "op", status=400, headers=lines) + + assert err.headers["X-Probe"] == f"{_PROBE}-first, {_PROBE}-second" + + +def test_no_fields_given_carries_none() -> None: + assert dict(CallError(CallErrorKind.REFUSED, "op", status=401).headers) == {} + + +def test_the_fields_are_never_rendered() -> None: + err = CallError(CallErrorKind.REFUSED, "op", status=401, headers={"X-Probe": _PROBE}) + + _assert_not_rendered(err) + assert _PROBE not in repr(err) + + +# --------------------------------------------------------------------------- +# Every site that builds a failure from an RPC answer, on both faces +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +def test_an_envelope_refusal_names_what_the_verifier_requires(face: Face) -> None: + answer = _Answer(401, _UNAUTHENTICATED, {**_JSON, AcceptSignatureHeader: accept_signature()}) + + failure = _discover_fails(face, answer) + + assert failure.kind is CallErrorKind.REFUSED + assert failure.status == 401 + assert failure.headers[AcceptSignatureHeader] == accept_signature() + _assert_not_rendered(failure) + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +def test_a_draining_gateway_says_when_to_come_back(face: Face) -> None: + failure = _discover_fails(face, _Answer(503, b"draining", {"retry-after": "86400"})) + + assert failure.kind is CallErrorKind.UNREACHABLE + assert failure.status == 503 + assert failure.headers["Retry-After"] == "86400" + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +def test_a_strictly_refused_envelope_keeps_its_fields(face: Face) -> None: + envelope = json.dumps( + {"code": "unauthenticated", "message": "not signed", "bogus": True} + ).encode() + answer = _Answer(401, envelope, {**_JSON, AcceptSignatureHeader: accept_signature()}) + + failure = _discover_fails(face, answer, strict=True) + + assert failure.kind is CallErrorKind.MALFORMED + assert failure.code == "unauthenticated" + assert failure.status == 401 + assert failure.headers[AcceptSignatureHeader] == accept_signature() + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +def test_a_refusal_in_a_coding_it_was_not_asked_for_keeps_its_fields(face: Face) -> None: + answer = _Answer(401, b"\x1f\x8b", {"content-encoding": "gzip", "x-probe": _PROBE}) + + failure = _discover_fails(face, answer) + + assert failure.kind is CallErrorKind.MALFORMED + assert failure.status == 401 + assert failure.headers["X-Probe"] == _PROBE + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +def test_a_refusal_past_the_read_cap_keeps_its_fields(face: Face) -> None: + answer = _Answer(401, b"x" * 64, {"x-probe": _PROBE}) + + failure = _discover_fails(face, answer, max_rpc_read_bytes=8) + + assert failure.kind is CallErrorKind.TOO_LARGE + assert failure.status == 401 + assert failure.headers["X-Probe"] == _PROBE + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +def test_a_refusal_nested_too_deep_keeps_its_fields(face: Face) -> None: + answer = _Answer(400, b"[" * 40 + b"]" * 40, {**_JSON, "x-probe": _PROBE}) + + failure = _discover_fails(face, answer) + + assert failure.kind is CallErrorKind.MALFORMED + assert failure.status == 400 + assert failure.headers["X-Probe"] == _PROBE + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +def test_a_refused_redirect_carries_no_fields(face: Face) -> None: + answer = _Answer( + 302, b"", {"location": f"https://elsewhere.test/next?token={_PROBE}", "x-probe": _PROBE} + ) + + failure = _discover_fails(face, answer) + + assert failure.kind is CallErrorKind.UNREACHABLE + assert failure.status == 302 + assert dict(failure.headers) == {} + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +def test_a_2xx_that_is_not_json_carries_no_fields(face: Face) -> None: + failure = _discover_fails(face, _Answer(200, b"", {"x-probe": _PROBE})) + + assert failure.kind is CallErrorKind.MALFORMED + assert dict(failure.headers) == {} + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +def test_a_strictly_refused_success_carries_no_fields(face: Face) -> None: + body = json.dumps({"ver": "1.0", "bogus": 1}).encode() + + failure = _discover_fails(face, _Answer(200, body, {**_JSON, "x-probe": _PROBE}), strict=True) + + assert failure.kind is CallErrorKind.MALFORMED + assert dict(failure.headers) == {} + + +def test_decode_still_takes_its_positional_arguments() -> None: + with pytest.raises(CallError) as caught: + decode("discover", 401, _UNAUTHENTICATED.decode(), ResourceResponse) + + assert caught.value.status == 401 + assert dict(caught.value.headers) == {} + + +# --------------------------------------------------------------------------- +# The delivery fetch +# --------------------------------------------------------------------------- + + +def _fetch_fails(face: Face, response: httpx.Response) -> CallError: + def respond(request: httpx.Request) -> httpx.Response: # noqa: ARG001 + return response + + transport = httpx.MockTransport(respond) + config = _config() + client: Any = ( + Client(config, http=httpx.AsyncClient(transport=transport)) + if face.name == "async" + else sync_client.Client(config, http=httpx.Client(transport=transport)) + ) + with pytest.raises(CallError) as caught: + face.run(client.fetch("https://edge.test/doc")) + return caught.value + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +def test_a_refused_proof_is_asked_for_again(face: Face) -> None: + failure = _fetch_fails( + face, + httpx.Response( + 401, + json={"error": "unauthorized", "reason": "pop_sig_invalid"}, + headers={AcceptSignatureHeader: POP_ACCEPT_SIGNATURE}, + ), + ) + + assert failure.status == 401 + assert failure.headers[AcceptSignatureHeader] == POP_ACCEPT_SIGNATURE + _assert_not_rendered(failure) + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +def test_an_edge_403_names_where_the_terms_are(face: Face) -> None: + rules = "https://publisher.test/.well-known/fora.json" + failure = _fetch_fails( + face, + httpx.Response( + 403, + json={"error": "forbidden"}, + headers={ContentRulesHeader: rules, ExchangeHeader: "exchange.test"}, + ), + ) + + hint = parse_discovery_hint(failure.status or 0, failure.headers) + assert (hint.content_rules, hint.content_rules_state) == (rules, "valid") + assert (hint.exchange, hint.exchange_state) == ("exchange.test", "valid") + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +def test_an_edge_redirect_carries_no_fields(face: Face) -> None: + failure = _fetch_fails( + face, + httpx.Response( + 302, + headers={"location": f"https://elsewhere.test/doc?token={_PROBE}", "x-probe": _PROBE}, + ), + ) + + assert failure.kind is CallErrorKind.UNREACHABLE + assert dict(failure.headers) == {} diff --git a/sdk/python/tests/test_client.py b/sdk/python/tests/test_client.py index 9340685e..64b7f2ce 100644 --- a/sdk/python/tests/test_client.py +++ b/sdk/python/tests/test_client.py @@ -855,6 +855,7 @@ def test_fetch_proves_possession_of_the_signers_key_not_a_second_one(face: Face) ) def test_a_rate_limit_is_handed_back_as_the_peer_sent_it(reset_at: str) -> None: from fora_sdk.client import _verbs + from fora_sdk.client._call import Answer plan = _verbs.Plan( op="discover", url="u", body=b"", headers={}, timeout=1.0, max_bytes=1 << 20, sent={} @@ -866,7 +867,9 @@ def test_a_rate_limit_is_handed_back_as_the_peer_sent_it(reset_at: str) -> None: "rate_limit": {"reset_at": reset_at, "remaining": 5}, } ) - result = _verbs.finish_discover(_verbs.ClientConfig(base_url="https://e.test"), plan, 200, body) + result = _verbs.finish_discover( + _verbs.ClientConfig(base_url="https://e.test"), plan, Answer(200, body) + ) assert result.rate_limit is not None assert result.rate_limit["reset_at"] == reset_at @@ -877,6 +880,7 @@ def test_a_rate_limit_is_handed_back_as_the_peer_sent_it(reset_at: str) -> None: # they said 300, and kept a vendor key they both dropped. def test_a_rate_limits_other_members_are_the_decoded_ones() -> None: from fora_sdk.client import _verbs + from fora_sdk.client._call import Answer plan = _verbs.Plan( op="discover", url="u", body=b"", headers={}, timeout=1.0, max_bytes=1 << 20, sent={} @@ -889,7 +893,9 @@ def test_a_rate_limits_other_members_are_the_decoded_ones() -> None: "rate_limit": {"limit": "300", "remaining": 299, "vendor_extra": "x"}, } ) - result = _verbs.finish_discover(_verbs.ClientConfig(base_url="https://e.test"), plan, 200, body) + result = _verbs.finish_discover( + _verbs.ClientConfig(base_url="https://e.test"), plan, Answer(200, body) + ) assert result.rate_limit is not None assert result.rate_limit["limit"] == 300, "a string-spelled int32 reached the caller" assert result.rate_limit["remaining"] == 299 diff --git a/sdk/python/tests/test_parity_corpora_nonempty.py b/sdk/python/tests/test_parity_corpora_nonempty.py index 3103120a..780316ba 100644 --- a/sdk/python/tests/test_parity_corpora_nonempty.py +++ b/sdk/python/tests/test_parity_corpora_nonempty.py @@ -56,6 +56,13 @@ def _whole(doc: Any) -> Any: _CORPUS_SPECS: list[tuple[pathlib.Path, Callable[[Any], Any], str]] = [ (GO_TESTDATA / "offer-verify-vectors.json", _vectors, "offer-verify"), (GO_TESTDATA / "acceptance-vectors.json", _vectors, "acceptance"), + (GO_TESTDATA / "acceptance-vectors.json", lambda d: d["refused"], "acceptance-refused"), + (GO_TESTDATA / "request-acceptance-vectors.json", _vectors, "request-acceptance"), + ( + GO_TESTDATA / "request-acceptance-vectors.json", + lambda d: d["refused"], + "request-acceptance-refused", + ), (GO_TESTDATA / "hashurl-vectors.json", _vectors, "hashurl"), (GO_TESTDATA / "multisig-chain-vectors.json", _vectors, "multisig-chain"), (GO_TESTDATA / "money-vectors.json", _vectors, "money"), @@ -115,6 +122,11 @@ def _whole(doc: Any) -> Any: lambda d: d["transport_failures"], "transport-failure", ), + ( + GO_CONNECT_TESTDATA / "refusal-header-vectors.json", + lambda d: d["refusals"], + "refusal-header", + ), ( GO_TESTDATA / "wire-names-vectors.json", lambda d: d["snake_from_json_name"], diff --git a/sdk/python/tests/test_refusal_header_parity.py b/sdk/python/tests/test_refusal_header_parity.py new file mode 100644 index 00000000..e8456a85 --- /dev/null +++ b/sdk/python/tests/test_refusal_header_parity.py @@ -0,0 +1,87 @@ +"""Replay of the refusal-header corpus (refusal-header-vectors.json), on both faces. + +The corpus is the cross-language contract for which failures carry the header fields of +the answer that produced them: an answer of status 400 or above, on the RPC legs and the +delivery fetch, and never a redirect or a 2xx the client could not read. Each row serves its +own answer through the real client and checks what ``CallError.headers`` carries, the +status beside it, and the discovery hint a caller parses from the two. +""" + +from __future__ import annotations + +from typing import Any + +import httpx +import pytest +from test_client import _IDS, FACES, Face, _config + +import fora_sdk.sync as sync_client +from conftest import GO_CONNECT_TESTDATA, load_json +from fora_sdk import parse_discovery_hint +from fora_sdk.client import CallError, Client + +_ROWS: list[dict[str, Any]] = load_json(GO_CONNECT_TESTDATA / "refusal-header-vectors.json")[ + "refusals" +] +_QUERY = {"exchange": "exchange.test", "uris": ["https://site.test/a"]} + + +def _serving(row: dict[str, Any]) -> httpx.MockTransport: + """An answer of exactly the row's status, content type, body and field lines.""" + + def respond(request: httpx.Request) -> httpx.Response: # noqa: ARG001 + fields = [("content-type", row["content_type"])] + [ + (name, value) for name, value in row["headers"] + ] + return httpx.Response(row["status"], headers=fields, content=row["body"].encode()) + + return httpx.MockTransport(respond) + + +def _failure(face: Face, row: dict[str, Any]) -> CallError: + transport = _serving(row) + config = _config(strict=row["strict"]) + client: Any = ( + Client(config, http=httpx.AsyncClient(transport=transport)) + if face.name == "async" + else sync_client.Client(config, http=httpx.Client(transport=transport)) + ) + with pytest.raises(CallError) as caught: + # Built inside the block: the blocking face runs the call as it is made. + face.run( + client.discover(dict(_QUERY)) + if row["leg"] == "rpc" + else client.fetch("https://edge.test/doc") + ) + return caught.value + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +@pytest.mark.parametrize("row", _ROWS, ids=[r["name"] for r in _ROWS]) +def test_a_failure_carries_what_the_corpus_records(face: Face, row: dict[str, Any]) -> None: + failure = _failure(face, row) + + if not row["carried"]: + # The whole set, not only the row's names: a failure that carries nothing carries + # nothing at all. + assert dict(failure.headers) == {} + else: + assert failure.status == row["status"] + for name, value in row["carried"].items(): + assert failure.headers[name] == value + assert value not in str(failure) + + hint = parse_discovery_hint(failure.status or 0, failure.headers) + assert { + "content_rules": hint.content_rules or "", + "content_rules_state": hint.content_rules_state, + "exchange": hint.exchange or "", + "exchange_state": hint.exchange_state, + } == row["hint"] + + +def test_the_corpus_pins_both_outcomes_on_both_legs() -> None: + seen = {(row["leg"], bool(row["carried"])) for row in _ROWS} + assert seen == {("rpc", True), ("rpc", False), ("content", True), ("content", False)} + assert any(row["strict"] for row in _ROWS) + assert any(row["hint"]["exchange_state"] == "valid" for row in _ROWS) diff --git a/sdk/python/tests/test_strict_requests.py b/sdk/python/tests/test_strict_requests.py new file mode 100644 index 00000000..fb431eb7 --- /dev/null +++ b/sdk/python/tests/test_strict_requests.py @@ -0,0 +1,366 @@ +"""``ClientConfig.strict_requests``: refuse a request the contract does not describe, before +it is signed. + +The generated models ignore a member they do not declare, so a plain dict carrying a +misspelled key passes the field-level check, is signed and sent, and the server drops the +member. A negative test whose mutation is that key then passes while testing nothing. +Under the switch a verb takes only its own generated model, or a RawBody that is sent +exactly as given, and the message as stamped must pass the published strict JSON Schema of +the request and the SDK's cross-field rules. + +Pinned on both faces, for every verb that takes a request message: a plain dict, or a model +of another message, is refused before anything is resolved, signed or sent; the verb's +model sends the same bytes with the switch on as with it off; a RawBody is sent unchecked. +Then what only the stamped message shows: a member the client adds itself (the configured +requester, a reflected offer), a cross-field rule the model does not check, and +``validation="off"``, which leaves the switch in force. +""" + +from __future__ import annotations + +import inspect +import json +from typing import TYPE_CHECKING, Any + +import pytest +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PrivateKey +from test_account import _account_config +from test_client import _IDS, FACES, REQUESTER, Face, Recorder, _config, _verified +from test_typed_inputs import _ASYNC, _CASES, _OUTSIDE_THE_CLAUSE, _SYNC, _answer, _Case +from wire.models import ( + DiscoveryRequest, + DisputeRequest, + DomainVerificationConfirmation, + DomainVerificationRequest, + GetAccountStatusRequest, + PushResourcesRequest, + RefreshCatalogRequest, + ResourceQuery, + SetReportingPolicyRequest, + SetTenantFeeRateRequest, +) + +import fora_sdk.sync as sync_client +from fora_sdk.client import AdminClient, CallError, CallErrorKind, ClientConfig, RawBody +from fora_sdk.core import sign_offer_jcs +from fora_sdk.resolvers import RegistrationRequirements + +if TYPE_CHECKING: + import httpx + from pydantic import BaseModel + +_FEE = {"tenant_id": "tenant-1", "fee_rate_bps": 250, "notes": "launch rate"} +_POLICY = { + "tenant_id": "tenant-1", + "required_fields": ["consumed_quantity"], + "window_seconds": 3600, +} +_ADMIN_CASES = [ + _Case("set_tenant_fee_rate", "AdminClient", SetTenantFeeRateRequest, {"rate": _FEE}), + _Case("set_reporting_policy", "AdminClient", SetReportingPolicyRequest, {"policy": _POLICY}), + _Case("request_domain_verification", "AdminClient", DomainVerificationRequest, + {"exchange": "exchange.test", "domain": "publisher.example"}), + _Case("confirm_domain_verification", "AdminClient", DomainVerificationConfirmation, + {"exchange": "exchange.test", "domain": "publisher.example", "token": "tok-1"}), +] +_ALL = [*_CASES, *_ADMIN_CASES] +_ALL_IDS = [f"{c.verb}-{i}" for i, c in enumerate(_ALL)] + +#: The admin setters answer the setting they stored, which their responses require. Every +#: other verb reads the shared answer. +_ANSWERS = { + "set_tenant_fee_rate": {"ver": "1.0", "rate": _FEE}, + "set_reporting_policy": {"ver": "1.0", "policy": _POLICY}, +} + +_ASYNC_FACES = {**_ASYNC, "AdminClient": AdminClient} +_SYNC_FACES = {**_SYNC, "AdminClient": sync_client.AdminClient} + +_URIS = ["https://site.test/a"] +#: The configured requester with a misspelled key, and how the refusal names it. +_MISSPELLED = {**REQUESTER, "domian": "agent.test"} +_NAMED = ( + "strict_requests refused the request at /requester: " + "Additional properties are not allowed ('domian' was unexpected)" +) + + +class _Probe: + """An endpoint resolver, a requirements reader and a pre-signing hook in one, each of + which records that it was reached. A refusal made before all of them leaves + ``reached`` empty: nothing was resolved, read or signed.""" + + def __init__(self) -> None: + self.reached: list[str] = [] + + def resolve_endpoint(self, host: str) -> str: + self.reached.append(f"resolve {host}") + return "https://exchange.test" + + def resolve_registration_requirements(self, exchange: str) -> RegistrationRequirements: + self.reached.append(f"read {exchange}") + return RegistrationRequirements() + + def before_sign(self, request: httpx.Request) -> httpx.Request: + self.reached.append("sign") + return request + + def config(self, **overrides: Any) -> ClientConfig: + return _config( + endpoint_resolver=self, + registration_requirements=self, + before_sign=self.before_sign, + **overrides, + ) + + +def _client(face: Face, face_class: str, config: ClientConfig, rec: Recorder) -> Any: + if face.name == "async": + return _ASYNC_FACES[face_class](config, http=rec.async_()) + return _SYNC_FACES[face_class](config, http=rec.sync()) + + +def _recorder(verb: str) -> Recorder: + return Recorder(_ANSWERS.get(verb) or _answer()) + + +def _send(face: Face, case: _Case, message: Any, config: ClientConfig) -> Recorder: + rec = _recorder(case.verb) + face.run(getattr(_client(face, case.face_class, config, rec), case.verb)(message)) + return rec + + +def _refused(face: Face, case: _Case, message: Any, config: ClientConfig) -> str: + """The refusal's text, after checking it is MALFORMED and that nothing was sent.""" + rec = _recorder(case.verb) + with pytest.raises(CallError) as caught: + face.run(getattr(_client(face, case.face_class, config, rec), case.verb)(message)) + assert caught.value.kind is CallErrorKind.MALFORMED + assert rec.seen == [] + return str(caught.value) + + +def _another(model: type[BaseModel]) -> BaseModel: + """A valid model of some other request message than ``model``.""" + other = RefreshCatalogRequest if model is GetAccountStatusRequest else GetAccountStatusRequest + return other(exchange="exchange.test") + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +@pytest.mark.parametrize("case", _ALL, ids=_ALL_IDS) +def test_a_plain_dict_is_refused_before_anything_is_resolved_signed_or_sent( + face: Face, case: _Case +) -> None: + probe = _Probe() + refusal = _refused(face, case, dict(case.message), probe.config(strict_requests=True)) + + assert refusal.endswith( + f"strict_requests refuses a plain dict; pass wire.models.{case.model.__name__}, " + "or RawBody to send a body exactly as given" + ) + assert probe.reached == [] + # The same dict with the switch off is signed and sent, so the probe's silence above + # is the refusal's doing. + control = _Probe() + assert len(_send(face, case, dict(case.message), control.config()).seen) == 1 + assert "sign" in control.reached + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +@pytest.mark.parametrize("case", _ALL, ids=_ALL_IDS) +def test_a_model_of_another_message_is_refused_the_same_way(face: Face, case: _Case) -> None: + other = _another(case.model) + probe = _Probe() + refusal = _refused(face, case, other, probe.config(strict_requests=True)) + + assert refusal.endswith( + f"strict_requests refuses a value of type {type(other).__name__}; " + f"pass wire.models.{case.model.__name__}, or RawBody to send a body exactly as given" + ) + assert probe.reached == [] + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +@pytest.mark.parametrize("case", _ALL, ids=_ALL_IDS) +def test_the_model_sends_the_same_bytes_with_the_switch_on_as_off( + face: Face, case: _Case +) -> None: + model = case.model.model_validate(case.message) + + on = _send(face, case, model, _account_config(strict_requests=True)) + off = _send(face, case, model, _account_config()) + + assert len(on.seen) == len(off.seen) == 1 + assert on.seen[0].url == off.seen[0].url + assert on.seen[0].content == off.seen[0].content + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +@pytest.mark.parametrize("case", _ALL, ids=_ALL_IDS) +def test_a_raw_body_is_sent_exactly_as_given(face: Face, case: _Case) -> None: + """Raw mode is how a test puts a message the SDK would never build on the wire, so the + switch does not reach it: no member is checked and none is added.""" + body = json.dumps({**case.message, "not_a_member": True}).encode() + + rec = _send(face, case, RawBody(body), _account_config(strict_requests=True)) + + assert [r.content for r in rec.seen] == [body] + + +def _call_naming_the_requester(face: Face, verb: str, config: ClientConfig, rec: Recorder) -> None: + """Run a verb that stamps the configured requester onto what it sends. A purchase + always does. A query does when its message names none, which a model can only leave + unset when it is built with ``model_construct``, skipping validation.""" + if verb == "discover": + query = ResourceQuery.model_construct(exchange="exchange.test", uris=_URIS) + face.run(face.client(config, rec).discover(query)) + elif verb == "resolve": + request = DiscoveryRequest.model_construct(query="jazz", uris=_URIS) + face.run(face.broker(config, rec).resolve(request)) + else: + offer, public = _offer() + face.run(face.client(config, rec).execute(_verified(public, offer), idempotency_key="i")) + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +@pytest.mark.parametrize("verb", ["discover", "resolve", "execute"]) +def test_an_unknown_member_of_the_configured_requester_is_refused_and_named( + face: Face, verb: str +) -> None: + """The requester is configuration the client stamps, not a model the caller built, so + only the check of the stamped message can see a key misspelled in it.""" + rec = Recorder(_answer()) + with pytest.raises(CallError) as caught: + _call_naming_the_requester( + face, verb, _config(requester=_MISSPELLED, strict_requests=True), rec + ) + assert caught.value.kind is CallErrorKind.MALFORMED + assert str(caught.value).endswith(_NAMED) + assert rec.seen == [] + # With the switch off the model ignores the key, and it goes out on the wire. + rec = Recorder(_answer()) + _call_naming_the_requester(face, verb, _config(requester=_MISSPELLED), rec) + assert rec.body()["requester"]["domian"] == "agent.test" + + +def _offer(**extra: Any) -> tuple[dict[str, Any], bytes]: + """A signed offer, with any extra members inside the signature, and its public key.""" + seed = bytes([7]) * 32 + offer: dict[str, Any] = { + "offer_id": "offer-1", + "exchange": "exchange.test", + "expires_at": "2099-01-01T00:00:00Z", + **extra, + } + signature, algorithm = sign_offer_jcs(seed=seed, offer=offer) + public = Ed25519PrivateKey.from_private_bytes(seed).public_key().public_bytes_raw() + return {**offer, "signature": signature, "signature_algorithm": algorithm}, public + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +def test_a_reflected_offer_carrying_a_member_this_sdk_does_not_know_is_refused( + face: Face, +) -> None: + """A purchase reflects each offer exactly as discovery returned it, so an offer from a + newer minor version carries its new member into the request. Its signature verifies; + the switch refuses the message all the same, naming where the member sits.""" + offer, public = _offer(future_member="x") + verified = _verified(public, offer) + + rec = Recorder(_answer()) + with pytest.raises(CallError) as caught: + face.run( + face.client(_config(strict_requests=True), rec).execute(verified, idempotency_key="i") + ) + assert caught.value.kind is CallErrorKind.MALFORMED + assert str(caught.value).endswith( + "strict_requests refused the request at /items/0/offer: " + "Additional properties are not allowed ('future_member' was unexpected)" + ) + assert rec.seen == [] + # With the switch off it is reflected as received, which is what a purchase does. + rec = Recorder(_answer()) + face.run(face.client(_config(), rec).execute(verified, idempotency_key="i")) + assert rec.body()["items"][0]["offer"]["future_member"] == "x" + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +def test_a_cross_field_rule_the_model_does_not_check_is_refused(face: Face) -> None: + """A catalog term carries its price (``resource_entry.terms.pricing_required``). The + generated model checks each field, not rules across them, so without the switch the + push is sent and only the server can refuse it.""" + case = _Case("push_resources", "CatalogClient", PushResourcesRequest, {}) + message = PushResourcesRequest.model_validate({ + "entries": [{"domain": "publisher.test", "path": "/x", + "terms": [{"semantics": "TERM_SEMANTICS_ENUMERATED"}]}], + "exchange": "exchange.test", + "tenant_id": "tenant-1", + }) + + refusal = _refused(face, case, message, _account_config(strict_requests=True)) + + assert refusal.endswith( + "strict_requests refused the request: cross-field rule(s) " + "resource_entry.terms.pricing_required" + ) + assert len(_send(face, case, message, _account_config()).seen) == 1 + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +def test_validation_off_leaves_the_switch_in_force(face: Face) -> None: + """``validation`` governs the generated model's check and nothing else: turning it off + neither lets a plain dict through nor skips the check of the stamped message.""" + case = _CASES[0] + assert case.verb == "discover" + + refusal = _refused( + face, case, dict(case.message), _account_config(strict_requests=True, validation="off") + ) + assert "strict_requests refuses a plain dict" in refusal + + query = ResourceQuery.model_construct(exchange="exchange.test", uris=_URIS) + config = _account_config(strict_requests=True, validation="off", requester=_MISSPELLED) + assert _refused(face, case, query, config).endswith(_NAMED) + + +def test_the_switch_is_off_by_default() -> None: + assert ClientConfig(base_url="https://exchange.test").strict_requests is False + + +@pytest.mark.parametrize("face", FACES, ids=_IDS) +def test_a_keyword_the_model_does_not_declare_is_dropped_before_the_switch_sees_it( + face: Face, +) -> None: + """The generated models ignore an undeclared member, the forward-compatible choice for + an answer, so a misspelled keyword never reaches the SDK and the request goes out + without it. A type checker reports the keyword, because the models are dataclass + transforms; that is where this mistake is caught. Pinned so the documented gap stays + the one described.""" + case = _Case("dispute", "Client", DisputeRequest, {}) + misspelled: dict[str, Any] = { + "descripton": "never arrived", + "exchange": "exchange.test", + "idempotency_key": "idem-2", + "reason": "DISPUTE_REASON_DELIVERY_FAILED", + "report_id": "r-1", + "transaction_id": "t-1", + } + + sent = _send(face, case, DisputeRequest(**misspelled), _account_config(strict_requests=True)) + + assert "descripton" not in sent.body() + assert "description" not in sent.body() + + +@pytest.mark.parametrize("faces", [_ASYNC_FACES, _SYNC_FACES], ids=["async", "sync"]) +def test_every_verb_that_takes_a_request_message_is_covered(faces: dict[str, Any]) -> None: + """A verb added to a client face without a case here would take a plain dict under the + switch untested. Every public verb is covered above or named as taking no message.""" + public = { + name + for cls in faces.values() + for name, member in vars(cls).items() + if not name.startswith("_") and inspect.isfunction(member) + } + + assert {case.verb for case in _ALL} | set(_OUTSIDE_THE_CLAUSE) == public diff --git a/sdk/python/tests/test_testing_any_requester.py b/sdk/python/tests/test_testing_any_requester.py new file mode 100644 index 00000000..a90e9e3c --- /dev/null +++ b/sdk/python/tests/test_testing_any_requester.py @@ -0,0 +1,202 @@ +"""``fora_sdk.testing``: acceptance signers that admit an empty requester. + +They must reproduce byte for byte what the shared corpora record for "a signer without the +check", answer exactly as the signers they twin for every other input, produce nothing the +SDK's verifiers accept, and stay out of the SDK: not re-exported from the root, and imported +by no module of it. +""" + +from __future__ import annotations + +import ast +import base64 +import pathlib +from typing import Any + +import pytest +from cryptography.hazmat.primitives.asymmetric.ed25519 import Ed25519PublicKey + +import fora_sdk +from conftest import GO_TESTDATA, load_json +from fora_sdk import testing +from fora_sdk.core import ( + jcs_acceptance_payload, + jcs_request_acceptance_payload, + sign_offer_acceptance_jcs, + sign_request_acceptance_jcs, + verify_offer_acceptance_jcs, + verify_request_acceptance_jcs, +) + +_OFFER = load_json(GO_TESTDATA / "acceptance-vectors.json") +_REQUEST = load_json(GO_TESTDATA / "request-acceptance-vectors.json") + + +def _requester(row: dict[str, Any]) -> dict[str, str]: + return { + "requester_id": row["requester_id"], + "requester_domain": row["requester_domain"], + "idempotency_key": row["idempotency_key"], + } + + +def _offer_fields(row: dict[str, Any]) -> dict[str, Any]: + return {"offer_sig": row["offer_sig"], **_requester(row)} + + +def _request_fields(row: dict[str, Any]) -> dict[str, Any]: + return {"items": [(i["offer_sig"], i["exchange"]) for i in row["items"]], **_requester(row)} + + +def _assert_the_recorded_signature_verifies(row: dict[str, Any], payload: bytes) -> None: + """The row's signature verifies over its bytes, so a refusal of them is a refusal of + the requester alone. Raises InvalidSignature otherwise.""" + public = Ed25519PublicKey.from_public_bytes(base64.b64decode(row["pubkey_b64"])) + public.verify(bytes.fromhex(row["signature_hex"]), payload) + + +@pytest.mark.parametrize("row", _OFFER["refused"], ids=[r["name"] for r in _OFFER["refused"]]) +def test_a_refused_offer_acceptance_is_reproduced(row: dict[str, Any]) -> None: + fields = _offer_fields(row) + payload = testing.jcs_acceptance_payload_any_requester(**fields) + signed = testing.sign_offer_acceptance_any_requester( + seed=bytes.fromhex(row["seed_hex"]), **fields + ) + + assert payload == row["canonical_jcs"].encode() + assert signed == (row["signature_hex"], "EdDSA") + _assert_the_recorded_signature_verifies(row, payload) + # The protocol's own canonicalizer and verifier refuse the same bytes. + with pytest.raises(ValueError, match="empty requester"): + jcs_acceptance_payload(**fields) + with pytest.raises(ValueError, match="empty requester"): + verify_offer_acceptance_jcs( + pubkey_b64=row["pubkey_b64"], signature_hex=row["signature_hex"], **fields + ) + + +@pytest.mark.parametrize("row", _REQUEST["refused"], ids=[r["name"] for r in _REQUEST["refused"]]) +def test_a_refused_request_acceptance_is_reproduced(row: dict[str, Any]) -> None: + fields = _request_fields(row) + payload = testing.jcs_request_acceptance_payload_any_requester(**fields) + signed = testing.sign_request_acceptance_any_requester( + seed=bytes.fromhex(row["seed_hex"]), **fields + ) + + assert payload == row["canonical_jcs"].encode() + assert signed == (row["signature_hex"], "EdDSA") + _assert_the_recorded_signature_verifies(row, payload) + with pytest.raises(ValueError, match="empty requester"): + jcs_request_acceptance_payload(**fields) + assert not verify_request_acceptance_jcs( + pubkey_b64=row["pubkey_b64"], signature_hex=row["signature_hex"], **fields + ) + + +def test_the_refused_rows_cover_both_requester_fields() -> None: + for doc in (_OFFER, _REQUEST): + assert {row["empty"] for row in doc["refused"]} == {"requester_id", "requester_domain"} + + +@pytest.mark.parametrize("row", _OFFER["vectors"], ids=[r["name"] for r in _OFFER["vectors"]]) +def test_the_offer_twins_are_the_signers_on_a_named_requester(row: dict[str, Any]) -> None: + fields = _offer_fields(row) + seed = bytes.fromhex(row["seed_hex"]) + + assert testing.jcs_acceptance_payload_any_requester(**fields) == ( + jcs_acceptance_payload(**fields) + ) + assert testing.sign_offer_acceptance_any_requester(seed=seed, **fields) == ( + sign_offer_acceptance_jcs(seed=seed, **fields) + ) + + +@pytest.mark.parametrize("row", _REQUEST["vectors"], ids=[r["name"] for r in _REQUEST["vectors"]]) +def test_the_request_twins_are_the_signers_on_a_named_requester(row: dict[str, Any]) -> None: + fields = _request_fields(row) + seed = bytes.fromhex(row["seed_hex"]) + + assert testing.jcs_request_acceptance_payload_any_requester(**fields) == ( + jcs_request_acceptance_payload(**fields) + ) + assert testing.sign_request_acceptance_any_requester(seed=seed, **fields) == ( + sign_request_acceptance_jcs(seed=seed, **fields) + ) + + +_NAMED = {"requester_id": "agent-1", "requester_domain": "agent.example", "idempotency_key": "i"} + + +@pytest.mark.parametrize( + "fields", + [ + {**_NAMED, "offer_sig": ""}, + # An unsigned offer is refused before the requester is looked at, by both. + {**_NAMED, "offer_sig": "", "requester_id": ""}, + ], +) +def test_every_other_offer_refusal_is_the_signers(fields: dict[str, Any]) -> None: + with pytest.raises(ValueError) as signer: + jcs_acceptance_payload(**fields) + with pytest.raises(ValueError) as twin: + testing.jcs_acceptance_payload_any_requester(**fields) + assert str(twin.value) == str(signer.value) + + +@pytest.mark.parametrize( + ("items", "requester"), + [ + ([], _NAMED), + ([("", "one.example")], _NAMED), + ([("sig", "")], _NAMED), + ([("sig", "one.example"), ("", "")], _NAMED), + ], +) +def test_every_other_request_refusal_is_the_signers( + items: list[tuple[str, str]], requester: dict[str, str] +) -> None: + with pytest.raises(ValueError) as signer: + jcs_request_acceptance_payload(items=items, **requester) + with pytest.raises(ValueError) as twin: + testing.jcs_request_acceptance_payload_any_requester(items=items, **requester) + assert str(twin.value) == str(signer.value) + + +def test_past_an_empty_requester_the_next_refusal_is_the_signers_next() -> None: + no_id = {**_NAMED, "requester_id": ""} + with pytest.raises(ValueError, match=r"^request acceptance requires at least one item$"): + testing.jcs_request_acceptance_payload_any_requester(items=[], **no_id) + + +def test_the_module_is_not_on_the_root() -> None: + for name in testing.__all__: + assert name not in fora_sdk.__all__ + assert not hasattr(fora_sdk, name) + + +def _imported_modules(path: pathlib.Path, package: str) -> set[str]: + """Every module ``path`` imports, relative imports resolved against ``package``.""" + found: set[str] = set() + for node in ast.walk(ast.parse(path.read_text(encoding="utf-8"))): + if isinstance(node, ast.Import): + found.update(alias.name for alias in node.names) + elif isinstance(node, ast.ImportFrom): + if node.level: + parts = package.split(".") + base = ".".join(parts[: len(parts) - node.level + 1]) + module = f"{base}.{node.module}" if node.module else base + else: + module = node.module or "" + found.add(module) + found.update(f"{module}.{alias.name}" for alias in node.names) + return found + + +def test_no_module_of_the_sdk_imports_it() -> None: + root = pathlib.Path(fora_sdk.__file__).parent + for path in root.rglob("*.py"): + if path.name == "testing.py" and path.parent == root: + continue + package = ".".join(("fora_sdk", *path.parent.relative_to(root).parts)) + imported = _imported_modules(path, package) + assert "fora_sdk.testing" not in imported, f"{path} imports fora_sdk.testing" diff --git a/sdk/ts/README.md b/sdk/ts/README.md index 3636dc12..0b490daa 100644 --- a/sdk/ts/README.md +++ b/sdk/ts/README.md @@ -47,16 +47,23 @@ const validate = new Ajv2020({ validateFormats: false }).compile(schema); Strict decoding uses these same files: `createClient(url, { strict: true })` refuses an answer that carries an unknown field or breaks a cross-field rule, and an error answer whose Connect envelope or `ErrorDetail` (binary `value` or `debug` projection) the -contract does not define; a refused envelope is `malformed` with `code` kept. A harness that tests -FORA services through the client has three more seams on it: `beforeSign` alters a +contract does not define; a refused envelope is `malformed` with `code`, `status` and `headers` kept. A harness that tests +FORA services through the client has four more seams on it: `beforeSign` alters a request just before it is signed, a `RawBody` passed in place of a verb's request is sent -exactly as given, and `ForaCallError.code` carries the Connect code of a refusal next to -its typed `detail`. `execute` returns each delivery URL as the Exchange issued it, and +exactly as given, `ForaCallError.code` carries the Connect code of a refusal next to its +typed `detail`, and `ForaCallError.headers` carries the header fields of a refusal (an +answer of status 400 or above, never a redirect), such as a verifier's `Accept-Signature` +or an edge's discovery headers for `parseDiscoveryHint(err.status ?? 0, err.headers)`. An injected send returns them as +`UnaryResponse.headers`; one that leaves them out yields refusals carrying none. `execute` returns each delivery URL as the Exchange issued it, and `fetch` dials it as given with the agent's proof of possession: the delivery edge verifies the URL, and its refusal comes back with the edge's `retrieval_auth_failure` reason. `createAdminClient` covers the operator RPCs, and `@fora-protocol/sdk/identity` mints an agent key, its key directory, the directory's response signatures -(`signDirectoryResponse`) and a signer. +(`signDirectoryResponse`) and a signer. `@fora-protocol/sdk/testing` signs what the +`./acceptance` signers refuse to: an acceptance naming an empty requester +(`signOfferAcceptanceAnyRequester`, `signRequestAcceptanceAnyRequester` and the two payload +functions), so a test can prove a verifier refuses it. Test support only; nothing in the SDK +imports it. Requests are signed under the Web Bot Auth profile of RFC 9421 (draft-ietf-webbotauth-httpsig-protocol-00). A signature covers `@method`, `@target-uri`, diff --git a/sdk/ts/client/content.ts b/sdk/ts/client/content.ts index baaec4fc..fdf40f67 100644 --- a/sdk/ts/client/content.ts +++ b/sdk/ts/client/content.ts @@ -15,7 +15,7 @@ import { retrievalAuthFailureDetail } from "../src/errordetail.ts"; import { escapePair } from "../src/host-ref.ts"; import { MAX_BODY_DEPTH, rawNestingDepth } from "../src/jsondepth.ts"; import { RequestIDHeader, SignatureAgentHeader } from "../src/wire.ts"; -import { IDENTITY_ENCODING, refuseUnrequestedEncoding } from "./transport.ts"; +import { answerHeaders, IDENTITY_ENCODING, refuseUnrequestedEncoding } from "./transport.ts"; import { requireScheme, skipSSRF, ssrfGuard } from "../resolvers/http.ts"; import { ForaCallError } from "./errors.ts"; import { concat, reclaim } from "./send.ts"; @@ -175,11 +175,17 @@ export async function fetchContent( kind: "unreachable", op, status: response.statusCode, + headers: answerHeaders(response.headers), cause: new Error("peer answered with a redirect, which this client does not follow"), }); } if (response.statusCode < 200 || response.statusCode >= 300) { - throw await edgeRefusal(op, response.statusCode, response.body); + throw await edgeRefusal( + op, + response.statusCode, + response.body, + answerHeaders(response.headers), + ); } const body = await readBody(op, response.body, maxBytes); return { @@ -343,12 +349,14 @@ async function edgeRefusal( op: string, status: number, body: AsyncIterable, + headers: Headers, ): Promise { const token = await edgeReason(body); const init = { kind: "refused" as const, op, status, + headers, ...(token !== "" ? { reason: token } : {}), }; if (token === "") return new ForaCallError(init); diff --git a/sdk/ts/client/errors.ts b/sdk/ts/client/errors.ts index 610ed10e..ab2ca865 100644 --- a/sdk/ts/client/errors.ts +++ b/sdk/ts/client/errors.ts @@ -56,11 +56,17 @@ export interface CallErrorInit { peerMessage?: string; /** The Connect code of the peer's answer, when one was decoded. */ code?: string; + /** The header fields of the answer the failure was built from. Kept only when `status` + * is an error status, 400 or above; see `ForaCallError.headers`. */ + headers?: Headers | undefined; /** The underlying failure, kept so a caller can still reach a custody or resolver * sentinel after the failure has been classified here. */ cause?: unknown; } +/** The status from which an answer is an error, and its header fields a refusal's. */ +const ERROR_STATUS = 400; + /** * ForaCallError is the client's typed failure. Every verb throws this and nothing else, * so `instanceof ForaCallError` is not a coin flip on the very type callers are told to @@ -91,6 +97,20 @@ export class ForaCallError extends Error { * refusals are edge tokens rather than Connect codes. `reason` may hold either kind of * token; this field holds only the Connect code. */ readonly code: string | undefined; + /** The header fields of the peer's error answer, 400 or above: an `Accept-Signature` + * naming what a verifier requires, a `Retry-After`, an edge's discovery headers + * (`parseDiscoveryHint(err.status ?? 0, err.headers)` reads them). Empty for any other + * failure — a local one, a transport one, a 2xx this client could not read, and a + * refused redirect, whose `Location` names a URL the peer chose and so must not reach a + * caller here. + * + * A copy; `get` is case-insensitive and joins a field sent on several lines with ", ". + * The values are the peer's, unvalidated, and never rendered into `message`. A value + * the Fetch `Headers` type cannot hold (a character above U+00FF) is left out. + * + * The rule is applied here, once: a site passes the answer's fields and this + * constructor keeps them only for an error status. */ + readonly headers: Headers; constructor(init: CallErrorInit) { super(renderCallError(init), init.cause !== undefined ? { cause: init.cause } : undefined); @@ -108,6 +128,10 @@ export class ForaCallError extends Error { // thing this field exists not to do. this.peerMessage = init.peerMessage ?? ""; this.code = init.code; + this.headers = + init.headers !== undefined && init.status !== undefined && init.status >= ERROR_STATUS + ? new Headers(init.headers) + : new Headers(); } /** diff --git a/sdk/ts/client/options.ts b/sdk/ts/client/options.ts index 4dee6e24..b7fc4c2e 100644 --- a/sdk/ts/client/options.ts +++ b/sdk/ts/client/options.ts @@ -150,7 +150,8 @@ export interface ClientOptions { * the Connect envelope may carry only code, message and details, must name a known * Connect code and carry well-formed details, and every ErrorDetail in it, binary * `value` and `debug` projection alike, must pass the strict ErrorDetail schema and the - * cross-field rules; a refused envelope keeps `code` and `status` and carries no detail. + * cross-field rules; a refused envelope keeps `code`, `status` and `headers` and carries + * no detail. * Defaults to false: the generated schemas drop unknown fields, which keeps a client * working against a newer Exchange. */ strict?: boolean; diff --git a/sdk/ts/client/send.ts b/sdk/ts/client/send.ts index 7d1cfbd1..7e96515f 100644 --- a/sdk/ts/client/send.ts +++ b/sdk/ts/client/send.ts @@ -30,6 +30,7 @@ import { Agent, type Dispatcher, request as undiciRequest } from "undici"; import { skipSSRF, ssrfGuard } from "../resolvers/http.ts"; import { ForaCallError } from "./errors.ts"; import { + answerHeaders, refuseUnrequestedEncoding, type UnaryRequest, type UnaryResponse, @@ -80,6 +81,7 @@ export function createUnarySend(opts: SendOptions): UnarySend { return { status: response.statusCode, body: await readBounded(response.body, req.maxBytes, req.op), + headers: answerHeaders(response.headers), }; } finally { reclaim(response.body); diff --git a/sdk/ts/client/strict-envelope.ts b/sdk/ts/client/strict-envelope.ts index d36b7cb3..74df2dcc 100644 --- a/sdk/ts/client/strict-envelope.ts +++ b/sdk/ts/client/strict-envelope.ts @@ -69,15 +69,16 @@ function firstUnknown(obj: Obj, known: ReadonlySet): string | undefined /** * checkStrictEnvelope refuses an error answer whose envelope the strict contract does not - * accept, as `malformed` naming the first failure. The refusal keeps `status` and `code`, - * the Connect code the lenient read reports, and carries no detail: the detail is part - * of what was refused. + * accept, as `malformed` naming the first failure. The refusal keeps `status`, `code` (the + * Connect code the lenient read reports) and the answer's header fields, and carries no + * detail: the detail is part of what was refused. */ export function checkStrictEnvelope( op: string, status: number, payload: unknown, code: string | undefined, + headers?: Headers, ): void { const problem = envelopeViolation(payload); if (problem === undefined) return; @@ -85,6 +86,7 @@ export function checkStrictEnvelope( kind: "malformed", op, status, + headers, cause: new Error(`strict decoding refused the error answer: ${problem}`), ...(code !== undefined ? { code } : {}), }); diff --git a/sdk/ts/client/transport.ts b/sdk/ts/client/transport.ts index f8186dd3..947b9866 100644 --- a/sdk/ts/client/transport.ts +++ b/sdk/ts/client/transport.ts @@ -17,7 +17,7 @@ import type { Window } from "../core/window.ts"; import { errorDetailFrom } from "../src/errordetail.ts"; import { MAX_BODY_DEPTH, rawNestingDepth } from "../src/jsondepth.ts"; import { parseWire, WireNamingError } from "../../../gen/ts/wire/base.ts"; -import { requireScheme } from "../resolvers/http.ts"; +import { requireScheme, undiciHeaderFields } from "../resolvers/http.ts"; import { ConnectProtocolVersion, ConnectProtocolVersionHeader, @@ -81,6 +81,9 @@ export interface UnaryRequest { export interface UnaryResponse { status: number; body: string; + /** The answer's header fields. A refusal carries them (`ForaCallError.headers`); a send + * that leaves them out yields a refusal carrying none. */ + headers?: Headers; } /** @@ -390,6 +393,7 @@ export function refuseUnrequestedEncoding( kind: "malformed", op, status, + headers: answerHeaders(headers), cause: new Error( `peer answered with content-encoding "${coding}" after being asked for identity; ` + "a coding this client did not negotiate cannot be read under a bound", @@ -397,6 +401,24 @@ export function refuseUnrequestedEncoding( }); } +/** + * answerHeaders turns undici's raw response headers into the Fetch `Headers` a failure + * carries. A field whose value `Headers` cannot hold — undici decodes values as UTF-8, and + * `Headers` refuses a character above U+00FF — is left out rather than thrown on, because + * building the description of a failure must never become a second failure. + */ +export function answerHeaders(record: Record): Headers { + const out = new Headers(); + for (const [name, value] of undiciHeaderFields(record)) { + try { + out.append(name, value); + } catch { + // A value outside the ByteString range: left out, as documented above. + } + } + return out; +} + /** * decodeResponse turns one answer into a parsed message, or throws the typed failure. * @@ -423,14 +445,15 @@ export function decodeResponse(op: string, response: UnaryResponse, strict = fal kind: "unreachable", op, status: response.status, + headers: response.headers, cause: new Error("peer answered with a redirect, which this client does not follow"), }); } const payload = parseJSON(op, response); if (response.status < 200 || response.status >= 300) { - const error = connectEnvelopeError(op, response.status, payload); + const error = connectEnvelopeError(op, response.status, payload, response.headers); if (strict && response.body.trim() !== "") { - checkStrictEnvelope(op, response.status, payload, error.code); + checkStrictEnvelope(op, response.status, payload, error.code, response.headers); } throw error; } @@ -452,6 +475,7 @@ function parseJSON(op: string, response: UnaryResponse): unknown { kind: "malformed", op, status: response.status, + headers: response.headers, cause: new Error(`answer nests deeper than ${MAX_BODY_DEPTH} containers`), }); } @@ -470,6 +494,7 @@ function parseJSON(op: string, response: UnaryResponse): unknown { kind: kindOfConnectCode(code), op, status: response.status, + headers: response.headers, reason: code, code, cause, @@ -479,6 +504,7 @@ function parseJSON(op: string, response: UnaryResponse): unknown { kind: "malformed", op, status: response.status, + headers: response.headers, cause, }); } @@ -488,6 +514,7 @@ function connectEnvelopeError( op: string, status: number, payload: unknown, + headers: Headers | undefined, ): ForaCallError { const envelope = isRecord(payload) ? payload : {}; // An envelope carrying no code is not a verdict the peer reached, so the STATUS decides @@ -514,6 +541,7 @@ function connectEnvelopeError( cause: typeof envelope["message"] === "string" ? envelope["message"] : undefined, code, + headers, }); } diff --git a/sdk/ts/package.json b/sdk/ts/package.json index 27e56439..b646346d 100644 --- a/sdk/ts/package.json +++ b/sdk/ts/package.json @@ -35,6 +35,7 @@ "./identity": "./core/identity.ts", "./hono": "./hono/middleware.ts", "./acceptance": "./src/acceptance.ts", + "./testing": "./src/testing.ts", "./resolvers": "./resolvers/index.ts", "./resolvers/edge": "./resolvers/edge.ts", "./client": "./client/index.ts" diff --git a/sdk/ts/resolvers/http.ts b/sdk/ts/resolvers/http.ts index 9f649d25..149cdb38 100644 --- a/sdk/ts/resolvers/http.ts +++ b/sdk/ts/resolvers/http.ts @@ -210,11 +210,25 @@ async function requestBounded( throw err; } const body = await readBounded(resp.body as AsyncIterable | null); - return boundedResponse(resp.statusCode, body, (name) => { - const value = resp.headers[name.toLowerCase()]; - if (value === undefined) return null; - return Array.isArray(value) ? value.join(", ") : value; - }); + const fields = undiciHeaderFields(resp.headers); + return boundedResponse(resp.statusCode, body, (name) => fields.get(name.toLowerCase()) ?? null); +} + +/** + * undiciHeaderFields reads undici's raw response headers as one value per field: the name + * lowercased, as undici already gives it, and a field sent on several lines joined with + * ", ", which is how a Fetch Headers object and the other two SDKs read it. An absent value + * is skipped. The one reading of undici's shape, shared by this tier and the client's. + */ +export function undiciHeaderFields( + record: Record, +): Map { + const fields = new Map(); + for (const [name, value] of Object.entries(record)) { + if (value === undefined) continue; + fields.set(name.toLowerCase(), Array.isArray(value) ? value.join(", ") : value); + } + return fields; } /** The SSRF-guarded default transport. The directory host is derived from a diff --git a/sdk/ts/src/acceptance-render.ts b/sdk/ts/src/acceptance-render.ts new file mode 100644 index 00000000..3641d787 --- /dev/null +++ b/sdk/ts/src/acceptance-render.ts @@ -0,0 +1,100 @@ +// The canonical acceptance renderer and the checks around it, shared by the protocol's +// acceptance signers (acceptance.ts) and the test signers (testing.ts). +// +// Deliberately not exported from the package manifest: none of this is an API of its +// own. The requester check is a parameter, run at the exact position the protocol's +// signer runs it, and every other check is this module's. acceptance.ts passes the check +// fora.proto requires of a signer; testing.ts passes one that admits an empty requester. +// One renderer and one copy of every other check serve both, so neither can drift. +// Mirror of Go internal/canonicalsign and the private renderers in Python's core. + +import canonicalize from "canonicalize"; + +import type { AcceptanceInput, RequestAcceptanceInput } from "./acceptance.ts"; +import { utf8Bytes } from "./base64url.ts"; + +/** Refuses a requester an acceptance may not name, given its id and domain, by throwing. */ +export type RequesterCheck = (requesterId: string, requesterDomain: string) => void; + +/** + * renderAcceptance reproduces the canonical signed bytes: + * JCS(protojson(AgentAcceptancePayload)) with every empty string field omitted. Throws on + * an empty offer signature, then on whatever `check` refuses. + */ +export function renderAcceptance(input: AcceptanceInput, check: RequesterCheck): Uint8Array { + if (input.offerSig === "") { + throw new Error( + "fora/acceptance: cannot accept an unsigned offer (empty offer signature)", + ); + } + check(input.requesterId, input.requesterDomain); + // proto omit-unpopulated: every empty string field is absent before JCS. The Go + // oracle gets that structurally from EmitUnpopulated=false; this record is + // hand-built, so the omission is applied once over the whole record rather than + // per key. The filter tests for the empty STRING, which covers every member + // AgentAcceptancePayload has (the field-set guard in the Go suite pins that list), so + // a string field added to the message cannot arrive without its omission. A + // non-string field would need its own zero-value test. + const payload: Record = { + offer_sig: input.offerSig, + requester_id: input.requesterId, + requester_domain: input.requesterDomain, + idempotency_key: input.idempotencyKey, + }; + const obj = Object.fromEntries(Object.entries(payload).filter(([, v]) => v !== "")); + const jcs = canonicalize(obj); + if (jcs === undefined) { + throw new Error("fora/acceptance: payload is not JSON-serializable"); + } + return utf8Bytes(jcs); +} + +/** Canonical JCS(protojson(...)) bytes for the complete ordered execute set. Throws on + * whatever `check` refuses, then on no items, then on an item with an empty offer + * signature or exchange. */ +export function renderRequestAcceptance( + input: RequestAcceptanceInput, + check: RequesterCheck, +): Uint8Array { + check(input.requesterId, input.requesterDomain); + if (input.items.length === 0) { + throw new Error("fora/acceptance: request acceptance requires at least one item"); + } + const items = input.items.map((item, index) => { + if (item.offerSig === "") { + throw new Error( + `fora/acceptance: request item ${index} has an empty offer signature`, + ); + } + if (item.exchange === "") { + throw new Error(`fora/acceptance: request item ${index} has an empty exchange`); + } + return { offer_sig: item.offerSig, exchange: item.exchange }; + }); + const payload: Record = { + items, + requester_id: input.requesterId, + requester_domain: input.requesterDomain, + idempotency_key: input.idempotencyKey, + }; + const obj = Object.fromEntries( + Object.entries(payload).filter(([, value]) => value !== ""), + ); + const jcs = canonicalize(obj); + if (jcs === undefined) { + throw new Error("fora/acceptance: request payload is not JSON-serializable"); + } + return utf8Bytes(jcs); +} + +/** signPayload signs canonical acceptance bytes with the agent's Ed25519 key and returns + * the hex signature an AgentAcceptance or AgentRequestAcceptance carries. */ +export async function signPayload( + payload: Uint8Array, + privateKey: CryptoKey, +): Promise { + const sig = new Uint8Array(await crypto.subtle.sign("Ed25519", privateKey, payload)); + let hex = ""; + for (const b of sig) hex += b.toString(16).padStart(2, "0"); + return hex; +} diff --git a/sdk/ts/src/acceptance.ts b/sdk/ts/src/acceptance.ts index 824af2a8..97d57912 100644 --- a/sdk/ts/src/acceptance.ts +++ b/sdk/ts/src/acceptance.ts @@ -17,9 +17,7 @@ // WebCrypto is the crypto primitive (the same one the L1 pop/verify faces use); // the caller supplies the imported CryptoKey, exactly as core/sign.ts does. -import canonicalize from "canonicalize"; - -import { utf8Bytes } from "./base64url.ts"; +import { renderAcceptance, renderRequestAcceptance, signPayload } from "./acceptance-render.ts"; /** The JOSE/JWA algorithm identifier advertised on AgentAcceptance.signature. Always EdDSA for Ed25519 * (mirror helpers.AcceptanceSignatureAlgorithm). */ @@ -56,12 +54,6 @@ function hexToBytes(hex: string): Uint8Array | undefined { return out; } -function bytesToHex(bytes: Uint8Array): string { - let hex = ""; - for (const b of bytes) hex += b.toString(16).padStart(2, "0"); - return hex; -} - /** * requireNamedRequester refuses an acceptance whose canonical bytes would name an empty * requester. Requester.id and Requester.domain are both REQUIRED, and bytes naming an @@ -89,32 +81,7 @@ function requireNamedRequester(requesterId: string, requesterDomain: string): vo * (fail-closed, mirror Go CanonicalAcceptanceBytes / Python jcs_acceptance_payload). */ export function acceptancePayload(input: AcceptanceInput): Uint8Array { - if (input.offerSig === "") { - throw new Error( - "fora/acceptance: cannot accept an unsigned offer (empty offer signature)", - ); - } - requireNamedRequester(input.requesterId, input.requesterDomain); - // proto omit-unpopulated: every empty string field is absent before JCS. The Go - // oracle gets that structurally from EmitUnpopulated=false; this record is - // hand-built, so the omission is applied once over the whole record rather than - // per key. Only idempotency_key can still arrive empty, because the requester - // fields are refused above, but the filter tests for the empty STRING, which - // covers every member AgentAcceptancePayload has (the field-set guard in the Go - // suite pins that list), so a string field added to the message cannot arrive - // without its omission. A non-string field would need its own zero-value test. - const payload: Record = { - offer_sig: input.offerSig, - requester_id: input.requesterId, - requester_domain: input.requesterDomain, - idempotency_key: input.idempotencyKey, - }; - const obj = Object.fromEntries(Object.entries(payload).filter(([, v]) => v !== "")); - const jcs = canonicalize(obj); - if (jcs === undefined) { - throw new Error("fora/acceptance: payload is not JSON-serializable"); - } - return utf8Bytes(jcs); + return renderAcceptance(input, requireNamedRequester); } /** @@ -125,9 +92,7 @@ export async function signOfferAcceptance( input: AcceptanceInput, privateKey: CryptoKey, ): Promise { - const payload = acceptancePayload(input); - const sig = new Uint8Array(await crypto.subtle.sign("Ed25519", privateKey, payload)); - return bytesToHex(sig); + return signPayload(acceptancePayload(input), privateKey); } /** @@ -174,44 +139,14 @@ export interface RequestAcceptanceInput { export function requestAcceptancePayload( input: RequestAcceptanceInput, ): Uint8Array { - requireNamedRequester(input.requesterId, input.requesterDomain); - if (input.items.length === 0) { - throw new Error("fora/acceptance: request acceptance requires at least one item"); - } - const items = input.items.map((item, index) => { - if (item.offerSig === "") { - throw new Error( - `fora/acceptance: request item ${index} has an empty offer signature`, - ); - } - if (item.exchange === "") { - throw new Error(`fora/acceptance: request item ${index} has an empty exchange`); - } - return { offer_sig: item.offerSig, exchange: item.exchange }; - }); - const payload: Record = { - items, - requester_id: input.requesterId, - requester_domain: input.requesterDomain, - idempotency_key: input.idempotencyKey, - }; - const obj = Object.fromEntries( - Object.entries(payload).filter(([, value]) => value !== ""), - ); - const jcs = canonicalize(obj); - if (jcs === undefined) { - throw new Error("fora/acceptance: request payload is not JSON-serializable"); - } - return utf8Bytes(jcs); + return renderRequestAcceptance(input, requireNamedRequester); } export async function signRequestAcceptance( input: RequestAcceptanceInput, privateKey: CryptoKey, ): Promise { - const payload = requestAcceptancePayload(input); - const sig = new Uint8Array(await crypto.subtle.sign("Ed25519", privateKey, payload)); - return bytesToHex(sig); + return signPayload(requestAcceptancePayload(input), privateKey); } export async function verifyRequestAcceptance( diff --git a/sdk/ts/src/testing.ts b/sdk/ts/src/testing.ts new file mode 100644 index 00000000..2c3a2f67 --- /dev/null +++ b/sdk/ts/src/testing.ts @@ -0,0 +1,57 @@ +// Acceptance signers that admit an empty requester, for a test proving that a verifier +// refuses them. +// +// fora.proto requires an acceptance to name its requester: "A signer refuses to sign bytes +// that name an empty requester, and a verifier refuses an acceptance whose canonical bytes +// name one, even when the signature over those bytes verifies." The signers in +// ./acceptance are that signer, so they cannot produce the one input a verifier's refusal +// is tested with. The four functions here are their twins with that single refusal +// skipped. They render through the same canonical form and keep every other refusal, so +// an acceptance they sign is valid in every respect but the requester, and a verifier's +// refusal of it is attributable to that alone. The `refused` lists of the shared +// acceptance-vectors.json and request-acceptance-vectors.json record exactly what they +// produce. +// +// Test support, published as @fora-protocol/sdk/testing: nothing in the SDK imports it, +// and an acceptance it signs is one a conforming party must never send. Go's helperstest +// and Python's fora_sdk.testing are its twins. WebCrypto only, so it runs wherever the +// acceptance signers do. + +import type { AcceptanceInput, RequestAcceptanceInput } from "./acceptance.ts"; +import { renderAcceptance, renderRequestAcceptance, signPayload } from "./acceptance-render.ts"; + +/** Admits every requester, an empty one included. Not exported, so nothing outside this + * module can hand it to the renderer the signers share. */ +function anyRequester(): void {} + +/** acceptancePayload (./acceptance) without its refusal of an empty requesterId or + * requesterDomain. */ +export function acceptancePayloadAnyRequester(input: AcceptanceInput): Uint8Array { + return renderAcceptance(input, anyRequester); +} + +/** signOfferAcceptance (./acceptance) without its refusal of an empty requesterId or + * requesterDomain. */ +export async function signOfferAcceptanceAnyRequester( + input: AcceptanceInput, + privateKey: CryptoKey, +): Promise { + return signPayload(acceptancePayloadAnyRequester(input), privateKey); +} + +/** requestAcceptancePayload (./acceptance) without its refusal of an empty requesterId + * or requesterDomain. */ +export function requestAcceptancePayloadAnyRequester( + input: RequestAcceptanceInput, +): Uint8Array { + return renderRequestAcceptance(input, anyRequester); +} + +/** signRequestAcceptance (./acceptance) without its refusal of an empty requesterId or + * requesterDomain. */ +export async function signRequestAcceptanceAnyRequester( + input: RequestAcceptanceInput, + privateKey: CryptoKey, +): Promise { + return signPayload(requestAcceptancePayloadAnyRequester(input), privateKey); +} diff --git a/sdk/ts/tests/acceptance-refusals.test.ts b/sdk/ts/tests/acceptance-refusals.test.ts new file mode 100644 index 00000000..0166cb1b --- /dev/null +++ b/sdk/ts/tests/acceptance-refusals.test.ts @@ -0,0 +1,88 @@ +// The refusals of the acceptance canonicalizers and signers, pinned: the exact text, and +// which refusal wins when an input has several faults. The renderer and the checks around +// it are shared with code that skips one of them (src/testing.ts), so the order and the +// wording are part of what must not move. Mirrors sdk/python/tests/test_acceptance_refusals.py. +import { describe, expect, it } from "vitest"; + +import { + type AcceptanceInput, + acceptancePayload, + type RequestAcceptanceInput, + requestAcceptancePayload, + signOfferAcceptance, + signRequestAcceptance, + verifyOfferAcceptance, + verifyRequestAcceptance, +} from "../src/acceptance.ts"; + +const UNSIGNED = "fora/acceptance: cannot accept an unsigned offer (empty offer signature)"; +const NO_ID = "fora/acceptance: acceptance names an empty requester: requesterId is empty"; +const NO_DOMAIN = "fora/acceptance: acceptance names an empty requester: requesterDomain is empty"; +const NO_ITEMS = "fora/acceptance: request acceptance requires at least one item"; + +const named = { requesterId: "agent-1", requesterDomain: "agent.example", idempotencyKey: "idem" }; + +async function keyPair(): Promise { + return (await crypto.subtle.generateKey({ name: "Ed25519" }, true, ["sign", "verify"])) as CryptoKeyPair; +} + +describe("offer acceptance refusals", () => { + const cases: [string, AcceptanceInput, string][] = [ + ["an unsigned offer", { ...named, offerSig: "" }, UNSIGNED], + ["no requester id", { ...named, offerSig: "sig", requesterId: "" }, NO_ID], + ["no requester domain", { ...named, offerSig: "sig", requesterDomain: "" }, NO_DOMAIN], + ["an unsigned offer before an empty requester", { ...named, offerSig: "", requesterId: "" }, UNSIGNED], + ["no id before no domain", { ...named, offerSig: "sig", requesterId: "", requesterDomain: "" }, NO_ID], + ]; + for (const [name, input, message] of cases) { + it(name, async () => { + const keys = await keyPair(); + expect(() => acceptancePayload(input)).toThrow(new Error(message)); + await expect(signOfferAcceptance(input, keys.privateKey)).rejects.toThrow(new Error(message)); + expect(await verifyOfferAcceptance(input, "00", keys.publicKey)).toBe(false); + }); + } +}); + +describe("request acceptance refusals", () => { + const item = { offerSig: "sig", exchange: "one.example" }; + const cases: [string, RequestAcceptanceInput, string][] = [ + ["no items", { ...named, items: [] }, NO_ITEMS], + [ + "an item with no offer signature", + { ...named, items: [{ offerSig: "", exchange: "one.example" }] }, + "fora/acceptance: request item 0 has an empty offer signature", + ], + [ + "an item naming no exchange", + { ...named, items: [{ offerSig: "sig", exchange: "" }] }, + "fora/acceptance: request item 0 has an empty exchange", + ], + [ + "the second item", + { ...named, items: [item, { offerSig: "", exchange: "one.example" }] }, + "fora/acceptance: request item 1 has an empty offer signature", + ], + ["no requester id", { ...named, requesterId: "", items: [item] }, NO_ID], + ["no requester domain", { ...named, requesterDomain: "", items: [item] }, NO_DOMAIN], + ["an empty requester before no items", { ...named, requesterId: "", items: [] }, NO_ID], + [ + "an empty requester before an empty item field", + { ...named, requesterDomain: "", items: [{ offerSig: "", exchange: "" }] }, + NO_DOMAIN, + ], + [ + "an empty offer signature before an empty exchange", + { ...named, items: [{ offerSig: "", exchange: "" }] }, + "fora/acceptance: request item 0 has an empty offer signature", + ], + ]; + for (const [name, input, message] of cases) { + it(name, async () => { + const keys = await keyPair(); + expect(() => requestAcceptancePayload(input)).toThrow(new Error(message)); + await expect(signRequestAcceptance(input, keys.privateKey)).rejects.toThrow(new Error(message)); + expect(await verifyRequestAcceptance(input, "00", keys.publicKey)).toBe(false); + }); + } +}); diff --git a/sdk/ts/tests/call-error-headers.test.ts b/sdk/ts/tests/call-error-headers.test.ts new file mode 100644 index 00000000..003c28ac --- /dev/null +++ b/sdk/ts/tests/call-error-headers.test.ts @@ -0,0 +1,290 @@ +// ForaCallError.headers: the header fields of a refusal, and only of a refusal. +// +// A peer's error answer says what it wants next in its header fields: a Web Bot Auth +// verifier names the signature it requires in Accept-Signature, a draining gateway sets +// Retry-After, a delivery edge's 403 carries the discovery headers. `headers` carries them +// for an answer of status 400 or above, on both legs. Every other failure carries none: a +// local one, a transport one, a 2xx this client could not read, and a refused redirect, +// whose Location names a URL the peer chose. +// +// The constructor owns the rule, so every case below is the same rule seen from a different +// site that builds the failure. Mirrors sdk/python/tests/test_call_error_headers.py. +import { createServer, type OutgoingHttpHeaders, type Server } from "node:http"; +import { describe, expect, it } from "vitest"; + +import { fetchContent } from "../client/content.ts"; +import { ForaCallError } from "../client/errors.ts"; +import { createUnarySend } from "../client/send.ts"; +import { answerHeaders, decodeResponse, unaryCall, type UnaryResponse } from "../client/transport.ts"; +import { acceptSignature } from "../core/wba.ts"; +import { parseDiscoveryHint } from "../src/discovery-hint.ts"; +import { POP_ACCEPT_SIGNATURE } from "../src/pop.ts"; +import { AcceptSignatureHeader, ContentRulesHeader, ExchangeHeader } from "../src/wire.ts"; + +const UNAUTHENTICATED = JSON.stringify({ code: "unauthenticated", message: "the request is not signed" }); +// A value no message carries by accident, so its absence from `message` is evidence. +const PROBE = "probe-6c1f0e"; + +function caught(run: () => unknown): ForaCallError { + try { + run(); + } catch (err) { + expect(err).toBeInstanceOf(ForaCallError); + return err as ForaCallError; + } + throw new Error("expected a ForaCallError"); +} + +async function caughtAsync(run: () => Promise): Promise { + const err = await run().then( + () => undefined, + (e: unknown) => e, + ); + expect(err).toBeInstanceOf(ForaCallError); + return err as ForaCallError; +} + +function expectNotRendered(err: ForaCallError): void { + for (const [name, value] of err.headers) { + expect(err.message, `the message renders ${name}`).not.toContain(value); + } +} + +function answer(status: number, body: string, fields: Record = {}): UnaryResponse { + return { status, body, headers: new Headers(fields) }; +} + +describe("the rule, at the constructor", () => { + for (const status of [undefined, 200, 204, 302, 304, 399]) { + it(`status ${String(status)} carries no fields`, () => { + const err = new ForaCallError({ + kind: "refused", + op: "op", + ...(status !== undefined ? { status } : {}), + headers: new Headers({ "X-Probe": PROBE }), + }); + expect([...err.headers]).toEqual([]); + }); + } + + for (const status of [400, 401, 403, 429, 503]) { + it(`status ${status} carries a copy of its fields`, () => { + const fields = new Headers({ "X-Probe": PROBE }); + const err = new ForaCallError({ kind: "refused", op: "op", status, headers: fields }); + fields.set("X-Probe", "changed"); + expect(err.headers.get("X-Probe")).toBe(PROBE); + // Read case-insensitively, as HTTP header fields are. + expect(err.headers.get("x-probe")).toBe(PROBE); + }); + } + + it("no fields given carries none", () => { + expect([...new ForaCallError({ kind: "refused", op: "op", status: 401 }).headers]).toEqual([]); + }); + + it("the fields are never rendered", () => { + const err = new ForaCallError({ + kind: "refused", + op: "op", + status: 401, + headers: new Headers({ "X-Probe": PROBE }), + }); + expectNotRendered(err); + }); +}); + +describe("answerHeaders", () => { + it("reads a field sent on several lines as one joined value", () => { + const headers = answerHeaders({ "x-probe": [`${PROBE}-first`, `${PROBE}-second`] }); + expect(headers.get("X-Probe")).toBe(`${PROBE}-first, ${PROBE}-second`); + }); + + it("leaves out a value the Headers type cannot hold, and keeps the rest", () => { + // undici decodes values as UTF-8; Headers refuses a character above U+00FF. + const headers = answerHeaders({ "x-euro": "price €", "x-probe": PROBE, "x-absent": undefined }); + expect(headers.get("x-euro")).toBeNull(); + expect(headers.get("x-absent")).toBeNull(); + expect(headers.get("x-probe")).toBe(PROBE); + }); +}); + +describe("every site that builds a failure from an RPC answer", () => { + it("an envelope refusal names what the verifier requires", () => { + const err = caught(() => + decodeResponse("discover", answer(401, UNAUTHENTICATED, { [AcceptSignatureHeader]: acceptSignature(false) })), + ); + expect(err.kind).toBe("refused"); + expect(err.status).toBe(401); + expect(err.headers.get(AcceptSignatureHeader)).toBe(acceptSignature(false)); + expectNotRendered(err); + }); + + it("a draining gateway says when to come back", () => { + const err = caught(() => decodeResponse("discover", answer(503, "draining", { "retry-after": "86400" }))); + expect(err.kind).toBe("unreachable"); + expect(err.headers.get("Retry-After")).toBe("86400"); + }); + + it("a strictly refused envelope keeps its fields", () => { + const envelope = JSON.stringify({ code: "unauthenticated", message: "not signed", bogus: true }); + const err = caught(() => + decodeResponse("discover", answer(401, envelope, { [AcceptSignatureHeader]: acceptSignature(false) }), true), + ); + expect(err.kind).toBe("malformed"); + expect(err.code).toBe("unauthenticated"); + expect(err.status).toBe(401); + expect(err.headers.get(AcceptSignatureHeader)).toBe(acceptSignature(false)); + }); + + it("a refusal nested too deep keeps its fields", () => { + const err = caught(() => decodeResponse("discover", answer(400, "[".repeat(40) + "]".repeat(40), { "x-probe": PROBE }))); + expect(err.kind).toBe("malformed"); + expect(err.status).toBe(400); + expect(err.headers.get("x-probe")).toBe(PROBE); + }); + + it("a refused redirect carries no fields", () => { + const err = caught(() => + decodeResponse( + "discover", + answer(302, "", { location: `https://elsewhere.test/next?token=${PROBE}`, "x-probe": PROBE }), + ), + ); + expect(err.kind).toBe("unreachable"); + expect(err.status).toBe(302); + expect([...err.headers]).toEqual([]); + }); + + it("a 2xx that is not JSON carries no fields", () => { + const err = caught(() => decodeResponse("discover", answer(200, "", { "x-probe": PROBE }))); + expect(err.kind).toBe("malformed"); + expect([...err.headers]).toEqual([]); + }); + + it("a send that returns no fields yields a refusal carrying none", () => { + const err = caught(() => decodeResponse("discover", { status: 401, body: UNAUTHENTICATED })); + expect(err.status).toBe(401); + expect([...err.headers]).toEqual([]); + }); +}); + +// --------------------------------------------------------------------------- +// Over a real socket, through the default send +// --------------------------------------------------------------------------- + +async function serving(status: number, body: string, headers: OutgoingHttpHeaders): Promise<[Server, number]> { + const server = createServer((_req, res) => { + for (const [name, value] of Object.entries(headers)) { + if (value !== undefined) res.setHeader(name, value); + } + res.writeHead(status); + res.end(body); + }); + await new Promise((done) => server.listen(0, "127.0.0.1", () => done())); + return [server, (server.address() as { port: number }).port]; +} + +async function withLoopback(run: () => Promise): Promise { + const saved = [process.env["SKIP_SSRF"], process.env["ALLOW_INSECURE"]]; + process.env["SKIP_SSRF"] = "true"; + process.env["ALLOW_INSECURE"] = "true"; + try { + return await run(); + } finally { + if (saved[0] === undefined) delete process.env["SKIP_SSRF"]; + else process.env["SKIP_SSRF"] = saved[0]; + if (saved[1] === undefined) delete process.env["ALLOW_INSECURE"]; + else process.env["ALLOW_INSECURE"] = saved[1]; + } +} + +function discoverAt(port: number): Promise { + return withLoopback(() => + unaryCall({ + op: "discover", + target: { baseURL: `http://127.0.0.1:${port}`, service: "fora.v1.ExchangeService", method: "DiscoverResources" }, + message: {}, + send: createUnarySend({ guarded: true }), + }), + ); +} + +describe("the RPC leg, over a socket", () => { + it("a refusal carries its Accept-Signature", async () => { + const [server, port] = await serving(401, UNAUTHENTICATED, { + "content-type": "application/json", + [AcceptSignatureHeader]: acceptSignature(false), + }); + const err = await caughtAsync(() => discoverAt(port)); + server.close(); + expect(err.status).toBe(401); + expect(err.headers.get(AcceptSignatureHeader)).toBe(acceptSignature(false)); + }); + + it("a field sent on two lines reads as one joined value", async () => { + const [server, port] = await serving(400, JSON.stringify({ code: "invalid_argument", message: "no" }), { + "content-type": "application/json", + "x-probe": [`${PROBE}-first`, `${PROBE}-second`], + }); + const err = await caughtAsync(() => discoverAt(port)); + server.close(); + expect(err.headers.get("X-Probe")).toBe(`${PROBE}-first, ${PROBE}-second`); + }); + + it("a refusal in a coding it was not asked for keeps its fields", async () => { + const [server, port] = await serving(401, "\u001f\u008b", { "content-encoding": "gzip", "x-probe": PROBE }); + const err = await caughtAsync(() => discoverAt(port)); + server.close(); + expect(err.kind).toBe("malformed"); + expect(err.status).toBe(401); + expect(err.headers.get("x-probe")).toBe(PROBE); + }); +}); + +describe("the delivery fetch, over a socket", () => { + async function fetchFrom(port: number): Promise { + const keyPair = (await crypto.subtle.generateKey({ name: "Ed25519" }, true, ["sign", "verify"])) as CryptoKeyPair; + return caughtAsync(() => + withLoopback(() => + fetchContent(`http://127.0.0.1:${port}/doc`, { keyPair, signatureAgent: "https://agent.test" }), + ), + ); + } + + it("a refused proof is asked for again", async () => { + const [server, port] = await serving(401, JSON.stringify({ error: "unauthorized", reason: "pop_sig_invalid" }), { + "content-type": "application/json", + [AcceptSignatureHeader]: POP_ACCEPT_SIGNATURE, + }); + const err = await fetchFrom(port); + server.close(); + expect(err.status).toBe(401); + expect(err.headers.get(AcceptSignatureHeader)).toBe(POP_ACCEPT_SIGNATURE); + expectNotRendered(err); + }); + + it("a 403 names where the terms are", async () => { + const rules = "https://publisher.test/.well-known/fora.json"; + const [server, port] = await serving(403, JSON.stringify({ error: "forbidden" }), { + "content-type": "application/json", + [ContentRulesHeader]: rules, + [ExchangeHeader]: "exchange.test", + }); + const err = await fetchFrom(port); + server.close(); + const hint = parseDiscoveryHint(err.status ?? 0, err.headers); + expect([hint.contentRules, hint.contentRulesState]).toEqual([rules, "valid"]); + expect([hint.exchange, hint.exchangeState]).toEqual(["exchange.test", "valid"]); + }); + + it("a redirect carries no fields", async () => { + const [server, port] = await serving(302, "", { + location: `https://elsewhere.test/doc?token=${PROBE}`, + "x-probe": PROBE, + }); + const err = await fetchFrom(port); + server.close(); + expect(err.kind).toBe("unreachable"); + expect([...err.headers]).toEqual([]); + }); +}); diff --git a/sdk/ts/tests/refusal-header.parity.test.ts b/sdk/ts/tests/refusal-header.parity.test.ts new file mode 100644 index 00000000..4f9b2bb3 --- /dev/null +++ b/sdk/ts/tests/refusal-header.parity.test.ts @@ -0,0 +1,129 @@ +// Replay of the refusal-header corpus (refusal-header-vectors.json), over real sockets. +// +// The corpus is the cross-language contract for which failures carry the header fields of +// the answer that produced them: an answer of status 400 or above, on the RPC legs and the +// delivery fetch, and never a redirect or a 2xx the client could not read. Each row serves +// its own answer from a real server, so the fields arrive through undici exactly as they +// would in production, and the replay checks what ForaCallError.headers carries, the status +// beside it, and the discovery hint a caller parses from the two. +import { createServer, type Server } from "node:http"; +import { describe, expect, it } from "vitest"; + +import { fetchContent } from "../client/content.ts"; +import { ForaCallError } from "../client/errors.ts"; +import { createUnarySend } from "../client/send.ts"; +import { unaryCall } from "../client/transport.ts"; +import { parseDiscoveryHint } from "../src/discovery-hint.ts"; +import vectorsFile from "../../go/connect/testdata/refusal-header-vectors.json"; + +interface Row { + name: string; + leg: string; + strict: boolean; + status: number; + content_type: string; + body: string; + headers: [string, string][]; + carried: Record; + hint: { content_rules: string; content_rules_state: string; exchange: string; exchange_state: string }; +} + +// Through unknown: the JSON module's inferred type keys each row's `carried` by that row's +// own field names, so no one Row type overlaps them all; the replay reads rows generically. +const rows = (vectorsFile as unknown as { refusals: Row[] }).refusals; + +async function serving(row: Row): Promise<[Server, number]> { + const server = createServer((_req, res) => { + // Grouped by name, so a repeated name goes out as separate field lines. + const lines = new Map(); + for (const [name, value] of row.headers) { + lines.set(name, [...(lines.get(name) ?? []), value]); + } + res.setHeader("content-type", row.content_type); + for (const [name, values] of lines) res.setHeader(name, values); + res.writeHead(row.status); + res.end(row.body); + }); + await new Promise((done) => server.listen(0, "127.0.0.1", () => done())); + return [server, (server.address() as { port: number }).port]; +} + +async function withLoopback(run: () => Promise): Promise { + const saved = [process.env["SKIP_SSRF"], process.env["ALLOW_INSECURE"]]; + process.env["SKIP_SSRF"] = "true"; + process.env["ALLOW_INSECURE"] = "true"; + try { + return await run(); + } finally { + if (saved[0] === undefined) delete process.env["SKIP_SSRF"]; + else process.env["SKIP_SSRF"] = saved[0]; + if (saved[1] === undefined) delete process.env["ALLOW_INSECURE"]; + else process.env["ALLOW_INSECURE"] = saved[1]; + } +} + +async function failure(row: Row): Promise { + const [server, port] = await serving(row); + try { + const err = await withLoopback(async () => { + if (row.leg === "rpc") { + return unaryCall({ + op: "discover", + target: { + baseURL: `http://127.0.0.1:${port}`, + service: "fora.v1.ExchangeService", + method: "DiscoverResources", + }, + message: {}, + send: createUnarySend({ guarded: true }), + strict: row.strict, + }); + } + const keyPair = (await crypto.subtle.generateKey({ name: "Ed25519" }, true, [ + "sign", + "verify", + ])) as CryptoKeyPair; + return fetchContent(`http://127.0.0.1:${port}/doc`, { keyPair, signatureAgent: "https://agent.test" }); + }).then( + () => undefined, + (e: unknown) => e, + ); + expect(err).toBeInstanceOf(ForaCallError); + return err as ForaCallError; + } finally { + server.close(); + } +} + +describe("refusal-header corpus", () => { + for (const row of rows) { + it(row.name, async () => { + const err = await failure(row); + if (Object.keys(row.carried).length === 0) { + // The whole set, not only the row's names: a failure that carries nothing + // carries nothing at all. + expect([...err.headers]).toEqual([]); + } else { + expect(err.status).toBe(row.status); + for (const [name, value] of Object.entries(row.carried)) { + expect(err.headers.get(name)).toBe(value); + expect(err.message).not.toContain(value); + } + } + const hint = parseDiscoveryHint(err.status ?? 0, err.headers); + expect({ + content_rules: hint.contentRules ?? "", + content_rules_state: hint.contentRulesState, + exchange: hint.exchange ?? "", + exchange_state: hint.exchangeState, + }).toEqual(row.hint); + }); + } + + it("pins both outcomes on both legs", () => { + const seen = new Set(rows.map((r) => `${r.leg} ${Object.keys(r.carried).length > 0}`)); + expect([...seen].sort()).toEqual(["content false", "content true", "rpc false", "rpc true"]); + expect(rows.some((r) => r.strict)).toBe(true); + expect(rows.some((r) => r.hint.exchange_state === "valid")).toBe(true); + }); +}); diff --git a/sdk/ts/tests/testing.parity.test.ts b/sdk/ts/tests/testing.parity.test.ts new file mode 100644 index 00000000..1e4be976 --- /dev/null +++ b/sdk/ts/tests/testing.parity.test.ts @@ -0,0 +1,264 @@ +// @fora-protocol/sdk/testing: acceptance signers that admit an empty requester. +// +// They must reproduce byte for byte what the shared corpora record for "a signer without +// the check", answer exactly as the signers they twin for every other input, produce +// nothing the SDK's verifiers accept, and stay out of the SDK: no source file imports +// them. Mirrors sdk/python/tests/test_testing_any_requester.py. +import { readdirSync, readFileSync, statSync } from "node:fs"; +import { join, relative, resolve } from "node:path"; +import { fileURLToPath } from "node:url"; +import ts from "typescript"; +import { describe, expect, it } from "vitest"; + +import { + acceptancePayload, + requestAcceptancePayload, + signOfferAcceptance, + signRequestAcceptance, + verifyOfferAcceptance, + verifyRequestAcceptance, +} from "../src/acceptance.ts"; +import { + acceptancePayloadAnyRequester, + requestAcceptancePayloadAnyRequester, + signOfferAcceptanceAnyRequester, + signRequestAcceptanceAnyRequester, +} from "../src/testing.ts"; +import acceptanceVectors from "../../go/helpers/testdata/acceptance-vectors.json"; +import requestVectors from "../../go/helpers/testdata/request-acceptance-vectors.json"; + +type Requester = { requester_id: string; requester_domain: string; idempotency_key: string }; +type Signed = { canonical_jcs: string; signature_hex: string; pubkey_b64: string; seed_hex: string }; +type OfferRow = Requester & Signed & { name: string; empty?: string; offer_sig: string }; +type RequestRow = Requester & + Signed & { name: string; empty?: string; items: { offer_sig: string; exchange: string }[] }; + +const offers = acceptanceVectors as { vectors: OfferRow[]; refused: OfferRow[] }; +const requests = requestVectors as unknown as { vectors: RequestRow[]; refused: RequestRow[] }; + +const PKCS8_ED25519_PREFIX = Uint8Array.from([ + 0x30, 0x2e, 0x02, 0x01, 0x00, 0x30, 0x05, 0x06, 0x03, 0x2b, 0x65, 0x70, 0x04, 0x22, 0x04, 0x20, +]); + +function hexToBytes(hex: string): Uint8Array { + const out = new Uint8Array(hex.length / 2); + for (let i = 0; i < out.length; i++) out[i] = Number.parseInt(hex.slice(i * 2, i * 2 + 2), 16); + return out; +} + +async function signingKey(seedHex: string): Promise { + const seed = hexToBytes(seedHex); + const pkcs8 = new Uint8Array(PKCS8_ED25519_PREFIX.length + seed.length); + pkcs8.set(PKCS8_ED25519_PREFIX, 0); + pkcs8.set(seed, PKCS8_ED25519_PREFIX.length); + return crypto.subtle.importKey("pkcs8", pkcs8, { name: "Ed25519" }, false, ["sign"]); +} + +async function verifyingKey(pubB64: string): Promise { + const raw = Uint8Array.from(atob(pubB64), (c) => c.charCodeAt(0)); + return crypto.subtle.importKey("raw", raw, { name: "Ed25519" }, false, ["verify"]); +} + +const text = (bytes: Uint8Array): string => new TextDecoder().decode(bytes); + +function offerInput(row: OfferRow) { + return { + offerSig: row.offer_sig, + requesterId: row.requester_id, + requesterDomain: row.requester_domain, + idempotencyKey: row.idempotency_key, + }; +} + +function requestInput(row: RequestRow) { + return { + items: row.items.map((i) => ({ offerSig: i.offer_sig, exchange: i.exchange })), + requesterId: row.requester_id, + requesterDomain: row.requester_domain, + idempotencyKey: row.idempotency_key, + }; +} + +// The row's signature verifies over its bytes, so a refusal of them is a refusal of the +// requester alone. +async function expectTheRecordedSignatureVerifies(row: Signed): Promise { + const ok = await crypto.subtle.verify( + "Ed25519", + await verifyingKey(row.pubkey_b64), + hexToBytes(row.signature_hex), + new TextEncoder().encode(row.canonical_jcs), + ); + expect(ok).toBe(true); +} + +describe("the refused rows are reproduced", () => { + for (const row of offers.refused) { + it(`offer acceptance ${row.name}`, async () => { + const input = offerInput(row); + expect(text(acceptancePayloadAnyRequester(input))).toBe(row.canonical_jcs); + expect(await signOfferAcceptanceAnyRequester(input, await signingKey(row.seed_hex))).toBe(row.signature_hex); + await expectTheRecordedSignatureVerifies(row); + // The protocol's own canonicalizer and verifier refuse the same bytes. + expect(() => acceptancePayload(input)).toThrow(/empty requester/); + expect(await verifyOfferAcceptance(input, row.signature_hex, await verifyingKey(row.pubkey_b64))).toBe(false); + }); + } + for (const row of requests.refused) { + it(`request acceptance ${row.name}`, async () => { + const input = requestInput(row); + expect(text(requestAcceptancePayloadAnyRequester(input))).toBe(row.canonical_jcs); + expect(await signRequestAcceptanceAnyRequester(input, await signingKey(row.seed_hex))).toBe(row.signature_hex); + await expectTheRecordedSignatureVerifies(row); + expect(() => requestAcceptancePayload(input)).toThrow(/empty requester/); + expect(await verifyRequestAcceptance(input, row.signature_hex, await verifyingKey(row.pubkey_b64))).toBe(false); + }); + } + + it("cover both requester fields", () => { + for (const doc of [offers, requests]) { + expect(new Set(doc.refused.map((r) => r.empty))).toEqual(new Set(["requester_id", "requester_domain"])); + } + }); +}); + +describe("on a named requester the twins are the signers", () => { + for (const row of offers.vectors) { + it(`offer acceptance ${row.name}`, async () => { + const input = offerInput(row); + const key = await signingKey(row.seed_hex); + expect(text(acceptancePayloadAnyRequester(input))).toBe(text(acceptancePayload(input))); + expect(await signOfferAcceptanceAnyRequester(input, key)).toBe(await signOfferAcceptance(input, key)); + }); + } + for (const row of requests.vectors) { + it(`request acceptance ${row.name}`, async () => { + const input = requestInput(row); + const key = await signingKey(row.seed_hex); + expect(text(requestAcceptancePayloadAnyRequester(input))).toBe(text(requestAcceptancePayload(input))); + expect(await signRequestAcceptanceAnyRequester(input, key)).toBe(await signRequestAcceptance(input, key)); + }); + } +}); + +describe("every other refusal is the signers'", () => { + const named = { requesterId: "agent-1", requesterDomain: "agent.example", idempotencyKey: "i" }; + const errorOf = (run: () => unknown): string => { + try { + run(); + } catch (err) { + return (err as Error).message; + } + throw new Error("expected a refusal"); + }; + for (const input of [ + { ...named, offerSig: "" }, + // An unsigned offer is refused before the requester is looked at, by both. + { ...named, offerSig: "", requesterId: "" }, + ]) { + it(`offer acceptance ${JSON.stringify(input)}`, () => { + expect(errorOf(() => acceptancePayloadAnyRequester(input))).toBe(errorOf(() => acceptancePayload(input))); + }); + } + for (const items of [ + [], + [{ offerSig: "", exchange: "one.example" }], + [{ offerSig: "sig", exchange: "" }], + [ + { offerSig: "sig", exchange: "one.example" }, + { offerSig: "", exchange: "" }, + ], + ]) { + it(`request acceptance ${JSON.stringify(items)}`, () => { + const input = { ...named, items }; + expect(errorOf(() => requestAcceptancePayloadAnyRequester(input))).toBe( + errorOf(() => requestAcceptancePayload(input)), + ); + }); + } + it("past an empty requester, the next refusal is the signer's next", () => { + expect(() => requestAcceptancePayloadAnyRequester({ ...named, requesterId: "", items: [] })).toThrow( + new Error("fora/acceptance: request acceptance requires at least one item"), + ); + }); +}); + +describe("the test signers stay out of the SDK", () => { + // An import is checked by the module it resolves to, the way the compiler resolves it + // under this package's tsconfig, as the Go and Python guards read imports with their + // languages' parsers. So an extensionless or .js specifier, the package's own name and + // a literal import() are all caught, and a comment or a string that merely names the + // module is not. Only an import() of a computed string is beyond a static check. + const root = fileURLToPath(new URL("..", import.meta.url)); + const tsconfig = ts.parseJsonConfigFileContent( + ts.readConfigFile(join(root, "tsconfig.json"), ts.sys.readFile).config, + ts.sys, + root, + ); + const testing = join(root, "src", "testing.ts"); + const renderer = join(root, "src", "acceptance-render.ts"); + + /** Whether source, written as file, imports module (an absolute path). */ + const imports = (file: string, source: string, module: string): boolean => + ts.preProcessFile(source, true, true).importedFiles.some(({ fileName }) => { + const resolved = ts.resolveModuleName(fileName, file, tsconfig.options, ts.sys).resolvedModule; + return resolved !== undefined && resolve(resolved.resolvedFileName) === module; + }); + + const sources = (dir: string): string[] => + readdirSync(join(root, dir)).flatMap((name) => { + const path = join(root, dir, name); + if (statSync(path).isDirectory()) return sources(relative(root, path)); + return name.endsWith(".ts") ? [relative(root, path)] : []; + }); + const files = ["src", "core", "client", "resolvers", "hono"].flatMap(sources); + const importsOf = (file: string, module: string): boolean => + imports(join(root, file), readFileSync(join(root, file), "utf8"), module); + + it("no source file imports ./testing", () => { + expect(tsconfig.errors).toEqual([]); + expect(files.length).toBeGreaterThan(0); + expect(files.filter((f) => importsOf(f, testing))).toEqual([]); + }); + + it("only the two signers import the shared renderer", () => { + expect(files.filter((f) => importsOf(f, renderer)).sort()).toEqual([ + join("src", "acceptance.ts"), + join("src", "testing.ts"), + ]); + }); + + describe("the check itself", () => { + const inSrc = join(root, "src", "probe.ts"); + const inClient = join(root, "client", "probe.ts"); + const bt = "`"; + + it.each([ + ["an extensionless specifier", inSrc, `import { a } from "./testing";`], + ["a .js specifier", inSrc, `import { a } from "./testing.js";`], + ["a .ts specifier", inSrc, `import { a } from "./testing.ts";`], + ["a path from another directory", inClient, `import { a } from "../src/testing";`], + ["a re-export", inSrc, `export * from "./testing.ts";`], + ["a side-effect import", inSrc, `import "./testing.ts";`], + ["a dynamic import", inSrc, `const m = await import("./testing.ts");`], + ["a dynamic import of a template literal", inSrc, `const m = await import(${bt}./testing.ts${bt});`], + ["a type-only import", inSrc, `import type { A } from "./testing.ts";`], + ["the package's own name", inSrc, `import { a } from "@fora-protocol/sdk-l1/testing";`], + ])("catches %s", (_, file, source) => { + expect(imports(file, source, testing)).toBe(true); + }); + + it.each([ + ["a line comment", `// import { a } from "./testing.ts";`], + ["a block comment", `/* import { a } from "./testing.ts"; */`], + ["a string", `const s = "./testing.ts";`], + ["the protocol's signers", `import { a } from "./acceptance.ts";`], + ["a dependency", `import { describe } from "vitest";`], + ])("ignores %s", (_, source) => { + expect(imports(inSrc, source, testing)).toBe(false); + }); + + it("catches an extensionless import of the shared renderer", () => { + expect(imports(inClient, `import { r } from "../src/acceptance-render";`, renderer)).toBe(true); + }); + }); +}); diff --git a/website/src/content/docs/components/agent-sdk/fetch-flow.mdx b/website/src/content/docs/components/agent-sdk/fetch-flow.mdx index e96aa708..5fba06b7 100644 --- a/website/src/content/docs/components/agent-sdk/fetch-flow.mdx +++ b/website/src/content/docs/components/agent-sdk/fetch-flow.mdx @@ -305,7 +305,8 @@ switching over a family of per-failure structs. type CallError struct { Kind CallErrorKind // the class; branch on this Op string // the verb that failed - Status int // HTTP status when the peer answered, 0 otherwise + Status int // HTTP status of the refusing answer; 0 when none arrived + Header http.Header // that answer's header fields, when its status is 400 or above Reason string // the peer's own refusal token when it sent one Code connectrpc.Code // the Connect code of the peer's answer; zero when none Detail *forav1.ErrorDetail // the peer's typed reason when it sent one @@ -333,14 +334,20 @@ type FetchError struct { Failure FetchFailure // FetchRefused, FetchUnreachable, FetchTooLarge, // FetchNotSignable, FetchMalformed Op string - Status int // HTTP status when the edge answered, 0 otherwise - Reason string // the edge's own refusal token when it sent one + Status int // HTTP status when the edge answered, 0 otherwise + Header http.Header // the header fields of the edge's error answer (400 or above) + Reason string // the edge's own refusal token when it sent one Err error } ``` Both carry `ReasonOf()`, which answers the peer's own token when it sent one and the -failure class otherwise. `connect.ErrorDetailFrom(err)` pulls the typed +failure class otherwise. `Header` holds what a refusal asks for next — a verifier's +`Accept-Signature`, a gateway's `Retry-After`, an edge's discovery headers, which +`helpers.ParseDiscoveryHint(err.Status, err.Header)` reads — and is nil for any failure +that is not an error answer, a refused redirect included. When an interceptor passed to +`connect.WithInterceptors` retries a call, `Status`, `Header` and `Code` describe the last +request it sent. `connect.ErrorDetailFrom(err)` pulls the typed `ErrorDetail` out of any error that carries one, including the `retrieval_auth_failure` the client synthesizes from a delivery edge's refusal token. `resolvers.ErrEndpointRefused` is a sentinel reachable through `errors.Is`. diff --git a/website/src/content/docs/components/agent-sdk/overview.mdx b/website/src/content/docs/components/agent-sdk/overview.mdx index 30c0300f..7a4d918a 100644 --- a/website/src/content/docs/components/agent-sdk/overview.mdx +++ b/website/src/content/docs/components/agent-sdk/overview.mdx @@ -62,8 +62,9 @@ an example does not follow fails a gate instead of reaching a reader. its verified offers and, for the rest, the reason they were refused. Failures across every verb arrive as one `connect.CallError` with a `CallErrorKind` -- `CallNotSent`, `CallRefused`, `CallUnreachable`, `CallMalformed`, `CallTooLarge`, `CallNotSignable` -- -plus the peer's own reason token, the Connect code of its answer (`Code`) and, when it -sent one, a typed detail reachable through `connect.ErrorDetailFrom`. +plus the peer's own reason token, the Connect code of its answer (`Code`), the status and +header fields of a refusal (`Status`, `Header`) and, when it sent one, a typed detail +reachable through `connect.ErrorDetailFrom`. ### TypeScript @@ -98,9 +99,12 @@ parsing responses itself or re-implementing request signing. | Pre-signing hook: change a request after the SDK built it, before it is signed | `connect.WithBeforeSign` | `beforeSign` in `ClientOptions` | `ClientConfig.before_sign` | | Raw mode: send a body exactly as given, still signed and decoded | `connect.WithRawBody` | `new RawBody(body)` in place of the request | `RawBody(body)` in place of the request | | Strict decoding: refuse an unknown field, a field-level or a cross-field violation, in a success answer or in an error envelope and its `ErrorDetail` | `connect.WithStrictDecoding` | `strict: true` | `ClientConfig(strict=True)` | +| Strict requests: refuse a plain dict, and a request with an unknown member or a cross-field violation, before it is signed | — (verbs take the generated messages) | — (verbs are typed with the request input types) | `ClientConfig(strict_requests=True)` | | The Connect code of a refusal, on its own field | `CallError.Code` | `ForaCallError.code` | `CallError.code` | +| The header fields of a refusal, such as `Accept-Signature` or `Retry-After`, and its status | `CallError.Header`, `CallError.Status` | `ForaCallError.headers`, `ForaCallError.status` | `CallError.headers`, `CallError.status` | | The operator RPCs: `AdminService` and domain verification | `connect.NewAdminClient` | `createAdminClient` | `AdminClient` | | Mint a throwaway agent: key, directory document, signer | `helpers.GenerateKey`, `helpers.DirectoryDocument`, `core.SigningTransportFor` | `generateKey`, `directoryDocument`, `signingTransportFor` | `generate_key`, `directory_document`, `signing_transport_for` | +| Sign an acceptance that names an empty requester, to test that a verifier refuses it | `helperstest.SignOfferAcceptanceAnyRequester`, `helperstest.SignRequestAcceptanceAnyRequester` | `signOfferAcceptanceAnyRequester`, `signRequestAcceptanceAnyRequester` (`./testing`) | `sign_offer_acceptance_any_requester`, `sign_request_acceptance_any_requester` (`fora_sdk.testing`) | | Read a document a party publishes and check it: the well-known manifest, the WBA key directory, the key revocation list, the license document | `resolvers.ReadManifest`, `resolvers.ReadWBADirectory`, `resolvers.ReadRevocationList`, `resolvers.ReadLicenseDocument` | `readManifest`, `readWBADirectory`, `readRevocationList`, `readLicenseDocument` | `read_manifest`, `read_wba_directory`, `read_revocation_list`, `read_license_document` | | Tell a missing document or a refused redirect from an outage | `resolvers.ErrDocumentNotFound`, `resolvers.ErrRedirectRefused` | `DocumentNotFound`, `RedirectRefused` | `DocumentNotFoundError`, `RedirectRefusedError` | | Check a message against the strict contract | `helpers.CheckStrict` | `checkStrict` | `check_strict` | @@ -124,9 +128,18 @@ A few rules hold in every language: only `code`, `message` and `details`, must name a known Connect code, and must carry well-formed details. Every `ErrorDetail` in it, the decoded binary `value` and the `debug` projection alike, is checked against the strict `ErrorDetail` schema and its - rules. A refused envelope is malformed, with the peer's Connect code still on the - error's code field and no detail. An empty or non-JSON error body comes from a gateway - rather than the service, so its HTTP status still classifies it. + rules. A refused envelope is malformed, with the peer's Connect code, HTTP status and + header fields still on the error, and no detail. An empty or non-JSON error body comes + from a gateway rather than the service, so its HTTP status still classifies it. +- **Strict requests are Python's.** A Python verb also takes a plain dict, and the + generated models ignore a member they do not declare, so a misspelled key would be + signed, sent and discarded by the server. `strict_requests` refuses the dict before + anything is resolved or signed, and checks the message as stamped, with the configured + requester and a purchase's reflected offers in it, against the published strict JSON + Schema of the request and the cross-field rules. `RawBody` is still sent exactly as + given. A Go verb takes the generated message, which has no member the proto does not + declare, and a TypeScript verb is typed with the request input types, so a misspelled + key in an object literal does not compile. - **A document reader returns the document or a typed error, never nothing.** By default each reader fetches through the SDK's guarded client, so the SSRF guard and the https-only scheme guard apply; `SKIP_SSRF` and `ALLOW_INSECURE` relax them for a @@ -164,6 +177,21 @@ A few rules hold in every language: - **Error details are read from the binary `value`.** A Connect error carries its typed reason as `details[].value`, the binary `ErrorDetail`. All three SDKs read it; the JSON SDKs fall back to the lowerCamelCase `debug` projection only when `value` is absent. +- **A refusal carries its answer's header fields, and only a refusal does.** An answer of + status 400 or above, on an RPC leg or the delivery fetch, hands its header fields to the + failure: a verifier's `Accept-Signature`, a gateway's `Retry-After`, an edge's discovery + headers, which the discovery-hint parser reads with the failure's status (0 when it has + none). A local failure, a transport failure, a 2xx the client could not read and a + refused redirect carry none; a redirect's `Location` names a URL the peer chose. The + fields are a copy, looked up by any spelling of the name, a field sent on several lines + joined with ", ", and never rendered into the message. Go keeps each line in its + `http.Header`. +- **The test signers are not protocol signers.** A protocol signer refuses to sign an + acceptance that names an empty requester, so a verifier's refusal of one can only be + tested with a signer that does not. The test-support packages hold twins of the four + acceptance functions that skip that refusal and keep every other one, reproducing the + `refused` rows of the shared acceptance corpora. Nothing in the SDK imports them, and an + acceptance they sign is one a conforming party must never send. - **The delivery edge checks the delivery URL.** A purchase returns each URL as the Exchange issued it, and fetch sends any URL it is given, so a harness can put any URL in front of an edge. The edge's refusal comes back with a `retrieval_auth_failure` diff --git a/website/src/content/docs/reference/changelog.mdx b/website/src/content/docs/reference/changelog.mdx index fc1788c3..273cee54 100644 --- a/website/src/content/docs/reference/changelog.mdx +++ b/website/src/content/docs/reference/changelog.mdx @@ -6,6 +6,86 @@ description: "FORA protocol changelog" This page contains curated public release notes. For the exhaustive maintainer and protocol history, see [`proto/CHANGELOG.md`](https://github.com/FORA-Protocol/protocol/blob/main/proto/CHANGELOG.md). +## Unreleased + +### Breaking against v1.1.0 + +**SDK users** + +- **Go client errors.** `connect.CallError` and `resolvers.FetchError` gain a `Header` + field, an `http.Header`, so neither struct compares with `==` any more. An RPC + `CallError` now carries the `Status` of the peer's error answer, which v1.1.0 left at 0, + so its message names that status, as the content leg's always did. + +### Changes since v1.1.0 + +**Three more SDK capabilities for conformance and e2e harnesses (no wire change).** A +harness can now read the header fields a refusal arrived with, sign an acceptance that +names an empty requester to prove that a receiver refuses it, and, in Python, have the +client refuse a request the contract does not describe before it is signed. + +- **Refusal headers.** A refusal carries the header fields of the answer that refused: + `CallError.headers` (Python, a mapping looked up case-insensitively), + `ForaCallError.headers` (TypeScript, a Fetch `Headers`) and `CallError.Header` with + `resolvers.FetchError.Header` (Go, an `http.Header`), on the RPC legs and the delivery + fetch alike. A Web Bot Auth verifier's `Accept-Signature`, a gateway's `Retry-After` and + an edge's discovery headers are read through the SDK; `parse_discovery_hint`, + `parseDiscoveryHint` and `helpers.ParseDiscoveryHint` read the last with the error's + status. Only an error answer, status 400 or above, carries fields: a local failure, a + transport failure, a 2xx the client could not read and a refused redirect carry none, + the redirect because its `Location` names a URL the peer chose. A strictly refused error + envelope keeps them, as it keeps its code. The fields are a copy and are never rendered + into the message. A field sent on several lines reads as one value + joined with ", " in Python and TypeScript; Go keeps the lines, and + `strings.Join(Header.Values(name), ", ")` reads the same value. Go's RPC + `CallError.Status` is now set beside them. TypeScript's `UnaryResponse` gains an optional + `headers`, which the default send fills; an injected send that leaves it out yields + refusals carrying none. The new `refusal-header-vectors.json` pins which failures carry + fields and what they read as, in all three SDKs. +- **Test-only acceptance signers.** A receiver must refuse an acceptance that names an + empty requester even when its signature verifies, and the SDK's own signers refuse to + produce one, so the test proving the refusal needs a signer that does. A test-support + package in each language holds twins of the four acceptance functions that skip only + that refusal: `helperstest.CanonicalAcceptanceBytesAnyRequester`, + `SignOfferAcceptanceAnyRequester`, `CanonicalRequestAcceptanceBytesAnyRequester` and + `SignRequestAcceptanceAnyRequester` (Go, package `sdk/go/helperstest`); + `jcs_acceptance_payload_any_requester`, `sign_offer_acceptance_any_requester`, + `jcs_request_acceptance_payload_any_requester` and + `sign_request_acceptance_any_requester` (Python, `fora_sdk.testing`, not re-exported + from `fora_sdk`); `acceptancePayloadAnyRequester`, `signOfferAcceptanceAnyRequester`, + `requestAcceptancePayloadAnyRequester` and `signRequestAcceptanceAnyRequester` + (TypeScript, export path `./testing`). Every other refusal is kept, in the same text + and order, and the protocol's signers and verifiers are unchanged and still refuse. + Each language renders both through one canonical renderer that takes the requester + check as a parameter: `sdk/go/internal/canonicalsign` in Go, private functions in + Python's `fora_sdk.core`, and `src/acceptance-render.ts` in TypeScript, which neither + package manifest exports. The twins reproduce the `refused` rows of + `acceptance-vectors.json` and `request-acceptance-vectors.json` byte for byte in all + three SDKs, and a test in each fails if production code imports the test package. +- **Strict requests (Python).** `ClientConfig(strict_requests=True)` refuses a request the + contract does not describe, before it is signed. The generated models ignore a member + they do not declare, so a misspelled key in a plain dict passes the default check, is + signed and sent, and is discarded by the server's JSON codec. Under the switch a verb + takes only its own generated request model or a `RawBody`: a plain dict, or a model of + another message, is refused as malformed before anything is resolved, read or sent, + naming the model to pass. The message as stamped, with `ver`, the configured + `requester` and a purchase's reflected offers in it, must then pass the published + strict JSON Schema of the request message and the cross-field rules, so an unknown + member at any depth is refused and named by its path. A `RawBody` is still sent exactly + as given. The switch applies whatever `validation` says and is off by default. A + keyword argument a generated model does not declare is dropped by the model before the + SDK sees it; a type checker reports it. Go verbs take the generated messages and + TypeScript verbs are typed with the request input types, so neither has the switch. + +**Go: a call an interceptor retries reports its last attempt (Go SDK fix; no wire +change).** An interceptor passed to `connect.WithInterceptors` may call `next` again to +retry a call. In v1.1.0 the client then kept what an earlier attempt had received: a call +whose last attempt got no answer reported its Connect code as the peer's +(`CallError.Code`), and under `WithStrictDecoding` an earlier attempt's error envelope could +be checked, and refused, in place of the last attempt's failure. Each request the call +sends is now recorded on its own, so `Code`, and the new `Status` and `Header`, describe +the last request sent. + ## v1.1.0 ### Breaking against v1.0.8 diff --git a/website/src/content/docs/reference/json-schemas.mdx b/website/src/content/docs/reference/json-schemas.mdx index 52a8f49f..ed542b5b 100644 --- a/website/src/content/docs/reference/json-schemas.mdx +++ b/website/src/content/docs/reference/json-schemas.mdx @@ -99,6 +99,12 @@ well-known manifest (`fora.v1.WellKnownManifest`), the WBA key directory [the agent SDK overview](/components/agent-sdk/overview/#testing-a-fora-service-through-the-sdk) for the readers in each language. +The Python client applies the check to what it sends as well, when it is built with +`ClientConfig(strict_requests=True)`. Each request, as the client stamped it and just +before it is signed, must pass the strict schema of its request message and the +cross-field rules, so an unknown member is refused rather than sent and discarded by the +server, and a verb refuses a plain dict in place of its generated request model. + ## What the schemas accept The schemas describe canonical proto-JSON as the FORA wire carries it. The two variants