Proposal: require canonical base64url encoding for the embedded signature (#247)

[chernistry]

What this changes

Section 3.2.2 already names the embedded signature field's encoding as
base64url, no padding. It does not say whether a different spelling of the
same bytes is the same record. This proposal adds one sentence that says it
isn't: the value must be canonically encoded, and a verifier must reject a
record whose signature is not.

This is a proposal, not a merge request for accepted text. Opened per the
request on #247 for a small, focused change with vectors and a scan of
existing data, rather than a bundle covering every related field.

What's open today

A signature over a Trust Record's canonical JSON form is 64 raw bytes for
the two algorithms records use in practice (Ed25519, and ES256 over P-256 as
raw r‖s); ES384, which section 3.2.1 also lists for JWT contexts, yields
96 bytes, 128 characters and no unused bits, so the question does not arise
there. Base64url without padding spends 86 characters (516 bits) encoding
512 real bits, so the final character carries 4 bits that no signature
algorithm produces. RFC 4648 section 3.5 requires those bits to be zero;
a decoder that only restores padding and decodes (rather than also checking
them) silently accepts a spelling where they aren't. Section 3.2.2 is silent
on which behavior is required, so two conformant-looking verifiers can
currently disagree about whether the same 64 bytes, spelled two different
ways, are the same record.

What this change does

• Spec (spec/trace-v0.2.md, section 3.2.2, embedded-signature bullet):
  adds one sentence requiring the canonical base64url spelling (RFC 4648
  section 3.5) and requiring a verifier to reject a non-canonical one.
• Schema (schema/trace-claim.json, its packaged copy at
  src/agentrust_trace/schema/trace-v0.2.json, and the v0.3 draft schema,
  which inherits signature from the canonical schema and picks up the
  same change when regenerated): tightens the signature pattern so an
  86-character value's final character must be A, Q, g, or w — the
  four base64url characters whose low 4 bits are zero. No other pattern in
  any of these files is touched.
• Model (src/agentrust_trace/models.py): the same tightened pattern on
  the signature field, plus an extended docstring.
• Validator (src/agentrust_trace/validate.py): the ECMA-to-Python
  pattern-translation registry updated to carry the new pattern string
  (required — the registry is keyed by exact pattern text, so a schema
  pattern change without a matching entry here would raise KeyError at
  validation time rather than silently doing the wrong thing).
• Docs (docs/schema.md): the signature field row gets the same
  canonical-encoding note.
• Vectors (examples/signature-encoding/): three signed Trust Records
  and a re-runnable scan script; see below.
• Tests (tests/test_signature_encoding.py,
  tests/test_adequacy_all_sets.py): recompute the vectors' claims from
  committed bytes rather than trusting them, and register the new vector
  set with the repo's vector-adequacy checks.

Scan: how many published signatures would stop passing

examples/signature-encoding/scan_published_signatures.py walks any number
of given roots, finds every 86-character value at the key signature, and
reports how many are already canonically encoded. It is not scoped to this
repo — pass it other checkouts to extend the count.

python examples/signature-encoding/scan_published_signatures.py \
  examples \
  /path/to/trace-tests-checkout-at-3af2b53 \
  /path/to/trace-registry-checkout

Run against this repository's own examples/, the agentrust-io/trace-tests
conformance corpus pinned at 3af2b53, and a read-only clone of
agentrust-io/trace-registry:

Root	Signatures found	Canonical	Non-canonical
examples/ (this repo)	224	222	2
trace-tests @ 3af2b53	6	6	0
trace-registry	4	4	0
Total	234	232	2

The 2 non-canonical values are not pre-existing records: they are this
proposal's own two deliberately non-canonical example vectors (below),
counted because the script scans examples/ unconditionally. Excluding
those 3 new files (1 canonical reuse + 2 non-canonical), 231 signatures
predate this change, and all 231 are already canonical. Under this
proposal, none of them would stop passing.

The 216 I gave on #247 on 22 September for the first two roots was an
undercount: the same two roots hold 227 signatures, all canonical, so the
conclusion there stands.

Vectors

examples/signature-encoding/ adds:

• 01-canonical-signature.json — a signed Trust Record whose signature is
  canonically spelled. Reuses the record, trusted key, and signature already
  published in examples/verifier-compatibility/01-known-version-verified.json,
  so the set introduces no new signing key.
• 02-non-canonical-respelling-low-bit.json and
  03-non-canonical-respelling-all-bits.json — the same 64 signature bytes,
  respelled with the final character's unused bits set to 0001 and 1111
  respectively. Two rejecting vectors rather than one, so the boundary isn't
  covered by a single bad string a shortcut implementation could
  special-case.

All three carry a decoded_signature_hex; tests/test_signature_encoding.py
decodes each record.signature itself and asserts the raw bytes match across
all three, rather than trusting the declared value. It also confirms the raw
Ed25519 bytes verify cryptographically under every spelling, isolating the
encoding as the rejection cause rather than a broken signature.

See examples/signature-encoding/README.md for the full format and the
adequacy rationale.

Current behavior of both implementations

Checked against main at the base this branch was cut from (includes #390):

• This repo's reference library, unpatched, decodes a non-canonical
  respelling leniently — the unused bits are discarded during decode, not
  checked — and verifies it. Confirmed with a standalone reproducer against
  the current sign/verify_record path; fix: distinguish missing and malformed signatures #390 (missing vs. malformed
  signature handling) doesn't touch this, since a non-canonical spelling is
  neither missing nor malformed in the sense that PR addresses.
• The TypeScript verifier under review in feat(trust-record-verify): a TypeScript verifier for TRACE v0.2, scored against the Python one #376 already refuses the same
  spelling, reporting signature_malformed.

So the two implementations already disagree, today, before this proposal.
Tightening the schema pattern (this PR) makes the reference library agree
with that refusal — through the schema-validation step both implementations
already run every record through first, rather than through a new check
specific to this one field.

Out of scope (follow-ups, not folded into this proposal)

• cnf.jwk.x and other embedded public-key coordinates raise the same
  canonical-encoding question and are not covered here.
• Trusted-key entries in verifier configuration have the same open question.
• sig.value in schema/trace-revocation.json /
  schema/trace-revocation-bundle.json carries the same base64url pattern
  and the same question, independently noticed while scoping this change.
  Its pattern is untouched by this PR.

Change classification

This PR doesn't set MAJOR/MINOR/PATCH for itself — that's the maintainer's
call. The scan above is meant to give that decision an evidence base: as far
as this scan reaches, no currently-published signature stops validating.

Type of change

• Editorial (typo, link fix, clarification: no normative effect)
• Non-breaking spec change (new optional field, new platform profile, informative addition)
• Breaking spec change (requires 14-day comment period and Project Lead sign-off)
• Schema change
• Example addition

Breaking or non-breaking is left to you; see "Change classification" above.

Spec section

§3.2.2 (Mandatory signature and freshness binding), the embedded-signature bullet.

Checklist

• DCO sign-off on all commits (git commit -s)
• CHANGELOG.md updated
• Change marked with <!-- CHANGED: #247 - embedded signature MUST be canonical base64url --> in spec text
• Backward compatibility statement included (for breaking changes): the scan above is the evidence; classification is yours

Context

Requested by @imran-siddique on #247, who asked for a small, focused
proposal rather than a bundle. Proposed by @chernistry.

[imran-siddique]

Alex, classify this as an incompatible tightening of the draft’s acceptance rules. I support requiring canonical encoding. Please document which previously accepted spellings are rejected and how existing records should be handled. Keep the scan’s conclusion limited to the repositories and revisions examined. The normative wording will need a maintainer to carry it; this scope decision is not implementation approval.

[imran-siddique]

@chernistry c07fc772 covers the three asks: the classification, the one affected length and its 15 rejected spellings, handling of existing records including the parent_record_hash consequence, and a scan scoped to the revisions read. Main has since landed #418, which makes the decoder refuse non-canonical spellings, and #412, which moved schema regex semantics onto a shared helper. models.py and validate.py now conflict. Please rebase onto main and route the new signature pattern through the #412 helper; the schema, spec sentence and vectors stay as they are. The normative sentence still needs a maintainer to carry it, and I will do that once the rebase is green.