Repository navigation
Checklist step position is graded in practice while signed-requests.yaml says it is informational #7577
Description
Summary
signed-requests.yaml states twice that grading is on the error code alone:
:56— "The checklist step number is informational; grading is on the stable error code only."
:199— "Grading requires byte-for-byte match on the error code; thefailed_stepvalue is informational."
We agree with that rule. The difficulty is that the corpus does not quite hold to it: every negative vector ships an expected_outcome.failed_step (values 0–12 plus 9a across the 28 negatives), and more importantly, checklist position becomes load-bearing whenever a request can fail more than one check — because then which code you return is decided by which step ran first.
So "grade on the stable error code" silently grades the verifier's internal ordering, which the same document says it does not.
Why this is more than a documentation nit
The ordering requirement propagates into implementations in a way the text does not suggest.
Our verifier delegates the checklist to a library (the spec's own SDK), passing it headers, the exchange, a capability, a JWKS resolver, a replay store and a revocation checker. The library runs steps 1–13 internally; we do not own the loop.
Counterparty discovery, however, happens before the library is invoked — it is a three-hop outbound walk (agent_url → capabilities → brand.json → jwks_uri → JWKS). When that walk fails we hold a failure but have no position in a sequence we are not running. Raising it where we discover it would make an unreachable counterparty outrank an earlier check, so a malformed Signature-Input would be reported as a discovery failure.
The only way to place the failure correctly is to wrap it in a resolver that throws when the library calls it — which happens to be step 7:
class _FailedDiscoveryJwksResolver:
def __call__(self, keyid):
raise SignatureVerificationError(self._code, step=7, ...)That construct exists solely to satisfy an ordering the spec text says is informational. We are not claiming it is wrong to require it — only that if it IS required, saying so would let implementers build for it deliberately rather than discover it from a failing vector.
What would make the rule hold
Each negative vector should carry exactly one fault, so its expected code is the same regardless of the order a conformant verifier evaluates checks in. Where a vector necessarily carries two (a malformed header on a request whose counterparty is also unresolvable), the expected outcome is a statement about precedence, and that precedence belongs in the normative text rather than implied by a fixture.
Concretely, one of:
- Audit the negative vectors for multi-fault cases and reduce each to a single fault. This keeps
:56's rule true as written and is the smallest change. - If checklist precedence IS normative, state it in
security.mdxas an ordering requirement, so it is implementable from the prose rather than inferred.failed_stepthen documents that requirement instead of being labelled informational. - Drop
failed_stepfrom the fixtures if it is genuinely unused, so its presence stops implying it is graded.
We would prefer (1), and (2) over the status quo — the current position is the only one where the requirement is real but undiscoverable.
Related
Same underlying shape as #7567: a vector that carries more than one fault cannot have an order-independent expected outcome. That issue is about invalid request bodies; this one is about checklist position. Both resolve the same way — one fault per vector.
Activity
KonstantinMirin commentedon Sep 18, 2026
Sharpening this, because the framing above understates it. The issue is not only that the corpus grades something the storyboard calls informational — it is that the spec specifies verifier control flow at all.
The spec mandates a sequence, not just outcomes
security.mdx:1077:and
:1010:So the numbered ordering and the short-circuit discipline are stated as requirements, and conformance is framed around steps rather than outcomes.
Why that is more than a style preference
A verifier's job, observably, is: given a request, either accept it or reject it with a specific code. That is the whole of what a counterparty can see and the whole of what interoperability depends on.
How an implementation arrives at that answer is not observable and should not be constrained. Resolution in particular is a distributed problem — an implementation may cache JWKS, may resolve counterparties through a separate service, may parallelise the discovery walk, may answer some checks from a warm cache before others that need a network hop. Requiring "short-circuit on first failure" over a fixed numbering forbids all of that, for no interoperability gain: the caller cannot tell the difference, and the emitted code is identical either way for any request with a single fault.
The spec already has the right shape elsewhere in the same document. It defines where a key is resolved from —
agent_url → capabilities → identity.brand_json_url → brand.json agents[] → jwks_uri → JWKS— which is a statement about where data lives, and is genuinely normative because both sides must agree on it. What it should add is which code is emitted when resolution fails at each point. What it should not add is the order in which a verifier evaluates unrelated checks.The concrete cost
We delegate the checklist to the SDK, so we do not own the loop. Counterparty discovery happens before the SDK is invoked. When that walk fails we hold a failure with no position in a sequence we are not running, and raising it where we discover it would make an unreachable counterparty outrank an earlier check.
The only way to place it correctly is to hand the SDK a resolver that throws when called, so the failure surfaces wherever the SDK happens to invoke it:
That object exists only to satisfy an ordering requirement. It encodes a step number into our code so that our internal control flow matches a numbering in the spec — which is exactly the coupling the spec should not be creating.
What we would suggest
failed_stepfrom the graded fixtures, or mark it clearly as non-normative in the fixture schema.If a particular SDK prefers a fixed evaluation order, that is a reasonable implementation choice for that SDK. It just should not become a conformance obligation for everyone implementing the protocol.
Same root as #7567: both are the corpus encoding an implementation assumption the protocol does not actually require.
bokelley commentedon Sep 18, 2026
Triage
Classification: Conformance failure (compliance corpus documentation gap)
Bucket(s): compliance-suite
Status: ready-to-implement
Milestone: Spec Backlog
What the experts said:
static/compliance/source/test-vectors/request-signing/negative/.failed_stepis schema-informal; ~10 early-step vectors carry placeholder bytes; regenerating valid signatures is optional and mechanical.My take: (Scope revised in response to @KonstantinMirin's follow-up.) The follow-up is correct on one point: the prior draft's
$commentapproach — annotating multi-fault vectors as "unreachable because step X fires first" — would itself encode an ordering assumption in the corpus, contradicting the stated goal. The right fix is Option 1 (reduce multi-fault vectors to single-fault), not document why secondary faults are ordered-away.ready-to-implementstatus is unchanged; implementation scope below is updated accordingly.Implementation scope:
static/compliance/source/test-vectors/request-signing/README.md— add a 6th "Adding vectors" checklist item: each vector MUST test exactly one failure mode; a request MUST NOT carry a secondary fault independently reachable via a different execution path, because multi-fault vectors implicitly grade verifier step ordering.$commentnotes asserting "unreachable because step X fires first" — that encodes the ordering assumption we are trying to remove. Vectors where two independent faults are structurally unavoidable represent unresolved normative-ordering questions; flag those for WG decision rather than quietly documenting them.docs/building/by-layer/L1/security.mdx— in the discovery-chain error table (~lines 1276–1284), add a cross-reference note on the 4 infrastructure codes pointing to the line-1558 suppression rule. Cross-reference only; no normative change..context/generate-test-vectors.mjsto make the single-fault invariant strictly hold.Triaged by Claude Code. Session: https://claude.ai/code/session_01GqJxKXDXLVFt7SG4gYtuQW
Generated by Claude Code
KonstantinMirin commentedon Sep 18, 2026
A security point, which we think is the stronger half of this
Beyond the control-flow question: we would question whether a seller should ever return anything other than a single "signature rejected" code.
There are 29 distinct
request_signature_*codes in the profile. Every one of them is returned to an unauthenticated caller — the signature IS the credential, so a request that fails verification is by definition from someone not yet established. The outcome is identical in all 29 cases: the request is refused. The only thing that varies is how much the refusal tells the caller about the seller's internal state.The spec already forbids this pattern elsewhere
security.mdx:280:security.mdx:294:That is exactly the reasoning we are applying here, stated normatively by the same document. It is hard to reconcile "never say you are not authorized for that account, that's an existence leak" with a mandate to distinguish
request_signature_key_unknownfromrequest_signature_key_revokedto an anonymous caller.What the taxonomy discloses
A few of the 29, as examples rather than an exhaustive list:
request_signature_key_unknownvsrequest_signature_key_revokedrequest_signature_brand_json_unreachable,request_signature_capabilities_unreachable,request_signature_jwks_unavailablerequest_signature_replayedrequest_signature_rate_abuserequest_signature_window_invalidvsrequest_signature_params_incompleteNone of these changes what a legitimate signer does next at runtime: the request was refused, and a correctly-configured signer does not hit any of them.
The obvious objection, and why we think it has a clean answer
The counter-argument is debuggability: a buyer with a skewed clock or a rotated key benefits from being told. That is a real cost and we do not want to wave it away — clock skew in particular is genuinely unpleasant to diagnose blind.
But the audience for that feedback is a buyer integrating, not a buyer in production traffic, and the right place to serve it is buyer-side conformance tooling — a test endpoint, a grader, a sandbox seller that deliberately answers verbosely. That keeps the diagnostic signal where it helps and out of the production refusal path, and it makes it the buyer conformance suite's concern rather than every seller's obligation.
Precedent for the split is ordinary: OAuth 2.0 Bearer (RFC 6750) defines three error codes for exactly this surface —
invalid_request,invalid_token,insufficient_scope— not a per-check taxonomy, and deliberately so.What we would suggest
request_signature_invalid(orrequest_signature_requiredfor the unsigned-but-required pre-check, since that one is a policy statement rather than a verification outcome and discloses nothing the capability document does not already publish).This also dissolves most of the ordering problem in the original post: with one code, which check ran first stops being observable, and the corpus no longer needs to encode a precedence the protocol never meant to require.
KonstantinMirin commentedon Sep 18, 2026
That reasoning is EXACTLY the problem. Spec should not mandate implementation and make any assumptions on the sequence of anything.
test(compliance): make 021 duplicate member identical; tighten README…
test(compliance): mirror step-0/1 request-signing negatives into prof…
test(compliance): make 021 duplicate member identical; tighten README…
test(compliance): complete profile 3.2 signing-vector mirrors
bokelley commentedon Oct 31, 2026
Part of this shipped: #8068 (3.3.0-beta.0) added the 'carry exactly one fault' rule to the request-signing vector README, mirrored the step-0/1 negatives into
profile-3.2/, and fixed the 3.2 corpus to it. (#7583 fixes a different issue, #7567.)Not decided or shipped, so per the WG rule ('anything not decided moves to 3.4') only the remainder moves to 3.4: the 14 placeholder-signature negatives still grade step order, and there is no normative condition-to-code or precedence text. Objections by Fri 2026-11-06. The milestone stays 3.3.0 for now; it will be moved after Nov 6.