fix(ci): node infra results, sign-in audit rows, keyed pseudonyms, run identity (D-1198, D-1276, D-1044, D-1103)

.github/wiki/guides/ci/Trust-Security-and-Privacy.md

@@ -402,7 +402,7 @@ Without a known-good backup and its KEK, this procedure cannot recover the chain
 
 What the chain does not give you: it detects an edit that was not recomputed. Someone who can write the database and recompute every later row defeats it; the TEAM checkpoint detects that (see Signed evidence, revocation and the audit checkpoint). A PERSONAL coordinator has neither a checkpoint nor a second writer to guard against, so the chain there is a tidy record, not evidence. Someone who can write the database can also clear heads and let the next `Append` re-chain them; that counts as a database writer too. (Removing the newest rows is caught by the seq check above, but only until the counter itself is edited.)
 
-Handoff for later Tickets: `roles.Authorize(ctx, actor, role, project)` and `roles.AuthorizeAction(ctx, actor, action, project)` are the calls a route makes; `WithAuthor(ctx, id)` carries a run's author for `run.approve`. Both write one `authorize` row per release- or admin-level decision (allowed or refused, an unknown role included) and none for member-level reads, so a mutation that writes its own row records two rows for one request. The `serve admin` commands write only their own mutation row and do not audit failed validation, failed token or certificate sign-in, or the local operator's own reads. The non-author rule for `run.approve` compares principal ids (an audit label such as `token:tok-1:bob` names `bob`); the local operator is exempt only on a PERSONAL coordinator, where it is the only person; on a TEAM coordinator it is refused as `self_approval` for a run it authored.
+Handoff for later Tickets: `roles.Authorize(ctx, actor, role, project)` and `roles.AuthorizeAction(ctx, actor, action, project)` are the calls a route makes; `WithAuthor(ctx, id)` carries a run's author for `run.approve`. Both write one `authorize` row per release- or admin-level decision (allowed or refused, an unknown role included) and none for member-level reads, so a mutation that writes its own row records two rows for one request. That two-row pattern is the design: the `authorize` row is the decision and exists whether or not the mutation follows (a refusal never reaches a mutation row), and the mutation row is written first and fails closed. A refused token or certificate sign-in writes a `signin.fail` row (`token:unknown`, `token:<id>:<principal>`, `user:<id>` or `certificate:unknown` as the actor, the reason key and the peer address in the detail, never the token). It is rate limited: one row per peer and kind per 30 seconds, at most 20 rows per 30 seconds across all peers, and the next row says how many were held back (`suppressed=N`). A refused `serve admin` mutation (a validation or role refusal) writes one `<action>.refused` row (`roles.grant.refused`, `roles.revoke.refused`, `token.issue.refused`, `token.revoke.refused`) with the reason key. The `serve admin` commands otherwise write only their own mutation row and do not audit the local operator's own reads. The non-author rule for `run.approve` compares principal ids (an audit label such as `token:tok-1:bob` names `bob`); the local operator is exempt only on a PERSONAL coordinator, where it is the only person; on a TEAM coordinator it is refused as `self_approval` for a run it authored.
 
 Wiring left to the coordinator process (P7-NODE-10 and P7-TRUST-26): start `audit.Log.RunSealer` (60 s) so rows other packages write get sealed, call `audit.SetScanner(run.Redactions)` before its first `Append` or `AppendTx` (the `serve admin` commands do this themselves), and resolve an Actor per request, not per connection. `store.CertSubjectPut` has no caller until the command that issues user certificates records their subject.
 
@@ -460,6 +460,10 @@ A node that is signing eligible signs the evidence of every job it runs. The sig
 
 A node signs only if its agent state directory holds an empty file named `signing-eligible` (a regular file, not a symlink, not writable by group or other). Create it, or let provisioning create it, only on a node whose jobs cannot read the node key: a separate job UID, a user namespace or a container. Without the file results carry no signature and count as unsigned. The coordinator's `signing_eligible` fact decides regardless of the file. A node key that was rotated away no longer verifies old evidence; a revoked key stays revoked (the id is derived from the key).
 
+The coordinator also checks the signature of a result frame before it settles the attempt from the frame. A frame that carries a signature the registry does not verify (unknown, revoked or wrong key, a node that did not hold the lease, a node that is not signing eligible, a replay) ends the attempt `lost`, which counts against the node and takes the infra retry; it never settles as a pass. An unsigned frame is still accepted, because a node without the marker file completes unsigned, and the release gate refuses unsigned remote evidence. A result a node reports as an infra failure, and a cancel the run did not ask for (a drain or an agent self-cancel), also end the attempt `lost` and take the infra retry; a cancel the run asked for settles as cancelled.
+
+The runner and coordinator ids in public provenance, evidence and statuses are pseudonyms: `runner-` or `coordinator-` and 12 hex characters of an HMAC-SHA256 under a secret the operator holds. The secret is 32 random bytes in `pseudonym.key` in the CI home (mode 0600, never a link), created by the first attempt or the first `nself ci artifacts sign` and never published; a wrong-sized, group-readable or non-regular file fails the attempt or the signing instead of falling back to an unkeyed hash. A dictionary of host names cannot recover a builder id without it. Deleting the file gives every id a new pseudonym.
+
 **Revocation.** `run.RevokeNode` (and `attest.Revoker`) revoke every evidence key a node pinned or ever used, in one transaction, and quarantine the runs those keys signed. The call returns the gates, posted statuses and promoted releases the revoked evidence supported. It writes an `evidence.revoke` audit event and one `release.quarantine` event per promoted release, and re-posts each affected status as `error` with the reason `node.revoked`. `nself ci artifacts verify` reports a quarantined release as `quarantined` and exits 1; it reads the quarantine index from the coordinator store, so on a machine without one it cannot know and reports nothing. A node revoked with `nodes revoke` (registry state `revoked`) or by `RevokeNode` no longer verifies new evidence; a key pinned for it later is refused too.
 
 **Audit checkpoint (TEAM only).** The audit chain detects an edit that was not recomputed. A signed checkpoint detects a database writer who recomputed every row.

free/ci/cmd/serve_v15.go

@@ -176,6 +176,7 @@ type pipelineFacts struct {
 	Trigger   string          `json:"trigger"`
 	CloneURL  string          `json:"clone_url"`
 	Synthetic json.RawMessage `json:"synthetic"`
+	Submitter string          `json:"submitter"` // the audit label of the user who submitted the run, when one did
 }
 
 // dispatchPipeline runs one queued trigger: clone the revision into a fresh directory, plan it and run the
@@ -206,6 +207,7 @@ func dispatchPipeline(home string, f *v15ServeFlags) coordinator.Dispatcher {
 		}
 		o := RunOptions{Root: dir, Home: home, Pipeline: row.Pipeline, Trigger: serveTrigger(facts.Trigger),
 			Container: !f.unsandboxed, ProviderRepo: providerRepoFor(row.Source, row.Project), Timeout: time.Duration(f.timeout) * time.Second, Stream: f.verbose}
+		o.User, o.Team = submitterIdentity(facts.Submitter, row.Project)
 		p, digest, subst, err := preparePlan(ctx, o)
 		if err != nil {
 			return coordinator.RunResult{}, fmt.Errorf("plan failed: %w", err)

free/ci/cmd/serve_v15_identity.go

@@ -0,0 +1,25 @@
+package main
+
+// Purpose: the user and team a served run counts against in the scheduler limits (D-1044).
+// Inputs:  the audit label the user API stored as the submitter of a run, and the run's project.
+// Outputs: the principal id (user) and the owner of the project (team).
+// Constraints: a run nobody submitted through the user API (a webhook, a provider delivery) has no user,
+//          and its team is still the project owner; an empty user and team keep the shared bucket.
+
+import (
+	"strings"
+
+	"github.com/nself-org/plugins/free/ci/internal/roles"
+)
+
+// submitterIdentity returns the user and team of a run. The team is the owner part of owner/repo: the
+// coordinator holds no team table, and the project owner is the organisation the repository belongs to.
+func submitterIdentity(submitter, project string) (user, team string) {
+	if strings.TrimSpace(submitter) != "" {
+		user = roles.PrincipalOf(submitter)
+	}
+	if owner, _, ok := strings.Cut(project, "/"); ok {
+		team = owner
+	}
+	return user, team
+}

free/ci/cmd/serve_v15_identity_test.go

@@ -0,0 +1,17 @@
+package main
+
+import "testing"
+
+func TestSubmitterIdentity(t *testing.T) {
+	for _, c := range []struct{ submitter, project, user, team string }{
+		{"user:Bob role=member", "acme/web", "bob", "acme"},
+		{"token:tok-1:mia role=admin", "acme/web", "mia", "acme"},
+		{"operator:cli:ali role=owner", "o/r", "ali", "o"},
+		{"", "acme/web", "", "acme"},
+		{"user:bob role=member", "noslash", "bob", ""},
+	} {
+		if u, tm := submitterIdentity(c.submitter, c.project); u != c.user || tm != c.team {
+			t.Errorf("submitterIdentity(%q, %q) = %q, %q; want %q, %q", c.submitter, c.project, u, tm, c.user, c.team)
+		}
+	}
+}

free/ci/internal/attest/frame.go

@@ -0,0 +1,42 @@
+package attest
+
+// Purpose: verifies the signature of a result frame (D-1103). The agent signs the frame's evidence under
+// the signed-remote payload (SignDocumentRaw) and sends the key id and signature beside it, not inside
+// it; the coordinator rebuilds the signed-remote document and checks it with RunnerVerifier, so the key,
+// purpose, revocation, lease holder, eligibility and replay rules are the evidence gate's own.
+// Inputs:  a ledger, the evidence bytes of the frame, and the frame's key id and signature text.
+// Outputs: nil, or a *RemoteError whose Reason says why (code E686).
+// Constraints: both signature fields are required (an unsigned frame is the caller's policy, not
+//          decided here); fail closed on every ledger error; an accepted signature is consumed.
+
+import (
+	"encoding/json"
+
+	"github.com/nself-org/plugins/free/ci/internal/model"
+)
+
+// VerifyFrame checks the signature value under keyID over the evidence of a result frame.
+func VerifyFrame(led func() Ledger, evidence []byte, keyID, value string) error {
+	if keyID == "" || value == "" {
+		return remoteFail(RemoteSignatureMissing, "no runner signature", nil)
+	}
+	obj, err := decodeObject(evidence)
+	if err != nil {
+		return remoteFail(RemoteSignatureInvalid, "no signing payload: "+err.Error(), nil)
+	}
+	producer, _ := obj["producer"].(map[string]any)
+	if producer == nil {
+		return remoteFail(RemoteSignatureInvalid, "evidence has no producer", nil)
+	}
+	producer["kind"] = "signed-remote"
+	obj["signature"] = map[string]any{"kind": "runner-ed25519", "key_id": keyID, "value": value}
+	doc, err := json.Marshal(obj)
+	if err != nil {
+		return remoteFail(RemoteSignatureInvalid, "no signing payload: "+err.Error(), nil)
+	}
+	var ev model.Evidence
+	if err = json.Unmarshal(doc, &ev); err != nil {
+		return remoteFail(RemoteSignatureInvalid, "the evidence does not decode: "+err.Error(), nil)
+	}
+	return (&RunnerVerifier{Ledger: led}).VerifyEvidence(doc, ev)
+}

free/ci/internal/attest/frame_test.go

@@ -0,0 +1,56 @@
+package attest_test
+
+// Purpose: the result-frame signature check (D-1103): a frame the node signed the way the agent does
+// verifies; a changed document, a wrong signer and a missing signature are each refused with their reason.
+
+import (
+	"encoding/base64"
+	"testing"
+
+	"github.com/nself-org/plugins/free/ci/internal/attest"
+)
+
+func TestVerifyFrame(t *testing.T) {
+	ev := attemptEvidence("att-1")
+	ev.Producer.Kind = "self-reported" // what a node's document says before the coordinator accepts the signature
+	doc := marshal(t, ev)
+	sign := func(d []byte, s seedSigner) (string, string) {
+		raw, err := attest.SignDocumentRaw(d, s)
+		if err != nil {
+			t.Fatal(err)
+		}
+		return attest.EvidenceKeyID(s.KeyID()), base64.RawStdEncoding.EncodeToString(raw)
+	}
+	keyID, value := sign(doc, keyA)
+	otherKey, otherValue := sign(doc, keyB)
+	changed := ev
+	changed.Revision = "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb"
+	tampered := marshal(t, changed)
+	for _, c := range []struct {
+		name     string
+		doc      []byte
+		key, sig string
+		ledger   func(*memLedger)
+		reason   attest.RemoteReason
+	}{
+		{name: "valid", doc: doc, key: keyID, sig: value},
+		{name: "changed document", doc: tampered, key: keyID, sig: value, reason: attest.RemoteRevisionMismatch},
+		{name: "signed by a node that did not hold the lease", doc: doc, key: otherKey, sig: otherValue, reason: attest.RemoteLeaseHolderMismatch},
+		{name: "revoked key", doc: doc, key: keyID, sig: value, ledger: func(l *memLedger) { l.revoked[keyID] = true }, reason: attest.RemoteKeyRevoked},
+		{name: "no key id", doc: doc, sig: value, reason: attest.RemoteSignatureMissing},
+		{name: "no signature", doc: doc, key: keyID, reason: attest.RemoteSignatureMissing},
+		{name: "not base64", doc: doc, key: keyID, sig: "not base64!", reason: attest.RemoteSignatureMalformed},
+	} {
+		l := newLedger()
+		if c.ledger != nil {
+			c.ledger(l)
+		}
+		err := attest.VerifyFrame(func() attest.Ledger { return l }, c.doc, c.key, c.sig)
+		if got := attest.RemoteReasonOf(err); got != c.reason {
+			t.Errorf("%s: reason %q (%v), want %q", c.name, got, err, c.reason)
+		}
+	}
+	if err := attest.VerifyFrame(func() attest.Ledger { return nil }, doc, keyID, value); attest.RemoteReasonOf(err) != attest.RemoteLedgerUnavailable {
+		t.Errorf("no ledger: %v, want ledger_unavailable", err)
+	}
+}

free/ci/internal/attest/profile.go

@@ -8,7 +8,9 @@ package attest
 // Outputs: Profile with RunnerID, CoordinatorID, Public, Text and URL.
 // Constraints: pure; fail closed: only the exact value Private turns the
 // profile off, so an empty or unknown visibility scrubs; a zero Profile
-// scrubs; pseudonyms are stable for one repo and differ between repos.
+// scrubs; pseudonyms are stable for one repo and differ between repos. With an operator key
+// registered (UsePseudonymKey) a pseudonym is an HMAC under it, so a dictionary of host names cannot
+// recover the id; without one it is the unkeyed hash.
 
 import (
 	"net/netip"
@@ -38,14 +40,15 @@ type RepoFacts struct {
 type Profile struct {
 	private bool
 	repoID  string
+	key     []byte // the operator's pseudonym key (pseudonym_key.go); nil keeps the unkeyed form
 	hosts   []string
 	users   []string
 	aliases map[string]string // literal -> replacement, longest applied first
 }
 
 // ProfileFor builds the profile of one repository.
 func ProfileFor(f RepoFacts) Profile {
-	return Profile{private: f.Visibility == Private, repoID: f.RepoID,
+	return Profile{private: f.Visibility == Private, repoID: f.RepoID, key: currentPseudonymKey(),
 		hosts: append([]string(nil), f.Hostnames...), users: append([]string(nil), f.Usernames...)}
 }
 
@@ -62,6 +65,9 @@ func (p Profile) pseudonym(kind, id string) string {
 	if p.private || id == "" {
 		return id
 	}
+	if len(p.key) > 0 { // keyed: a dictionary of host names cannot recover the id (D-1198)
+		return kind + "-" + keyedPseudonym(p.key, "nself-ci/"+kind+"/v2\x00"+p.repoID+"\x00"+id)
+	}
 	return kind + "-" + Digest([]byte("nself-ci/" + kind + "/v1\x00" + p.repoID + "\x00" + id))[:12]
 }
 

free/ci/internal/attest/pseudonym_key.go

@@ -0,0 +1,145 @@
+package attest
+
+// Purpose: the operator-held secret behind the publication pseudonyms (D-1198). An unkeyed hash of a
+// repo id and a host name lets anyone with a dictionary of host names recover the builder id of a
+// public provenance statement; an HMAC under a secret the operator keeps does not.
+// Inputs:  the CI home directory (LoadPseudonymKey) or a key (UsePseudonymKey).
+// Outputs: the key a Profile built afterwards uses for RunnerID and CoordinatorID.
+// Constraints: the key is 32 random bytes in <home>/pseudonym.key, mode 0600, never a link, created
+//          on first use; a file of another size, mode or type is an error, never replaced. With no
+//          key registered a Profile keeps the unkeyed pseudonym (the form the golden statements hold),
+//          so a caller that publishes must register one. The key never appears in a document or a log.
+
+import (
+	"crypto/hmac"
+	"crypto/rand"
+	"crypto/sha256"
+	"encoding/hex"
+	"errors"
+	"fmt"
+	"io"
+	"os"
+	"path/filepath"
+	"runtime"
+	"sync"
+	"sync/atomic"
+)
+
+// PseudonymKeyFile is the name of the key file under the CI home.
+const PseudonymKeyFile = "pseudonym.key"
+
+const pseudonymKeyLen = 32
+
+var (
+	pseudonymKey atomic.Pointer[[]byte]
+	loadMu       sync.Mutex // one creator at a time within the process
+)
+
+// UsePseudonymKey sets the key later Profiles pseudonymize with; nil clears it. A key shorter than 32
+// bytes is refused.
+func UsePseudonymKey(key []byte) error {
+	if key == nil {
+		pseudonymKey.Store(nil)
+		return nil
+	}
+	if len(key) < pseudonymKeyLen {
+		return errors.New("attest: the pseudonym key must be at least 32 bytes")
+	}
+	k := append([]byte(nil), key...)
+	pseudonymKey.Store(&k)
+	return nil
+}
+
+func currentPseudonymKey() []byte {
+	if k := pseudonymKey.Load(); k != nil {
+		return *k
+	}
+	return nil
+}
+
+// UsePseudonymKeyAt loads (creating it when absent) the key under home and registers it, so a process
+// that publishes without running an attempt (`nself ci artifacts sign`) pseudonymizes with the same key
+// as a run does.
+func UsePseudonymKeyAt(home string) error {
+	key, err := LoadPseudonymKey(home)
+	if err != nil {
+		return err
+	}
+	return UsePseudonymKey(key)
+}
+
+// LoadPseudonymKey returns the key under home, creating it when the file is absent.
+func LoadPseudonymKey(home string) ([]byte, error) {
+	if home == "" {
+		return nil, errors.New("attest: no CI home for the pseudonym key")
+	}
+	if err := os.MkdirAll(home, 0o700); err != nil {
+		return nil, err
+	}
+	path := filepath.Join(home, PseudonymKeyFile)
+	loadMu.Lock()
+	defer loadMu.Unlock()
+	key, err := readPseudonymKey(path)
+	if errors.Is(err, os.ErrNotExist) {
+		return createPseudonymKey(path)
+	}
+	return key, err
+}
+
+func readPseudonymKey(path string) ([]byte, error) {
+	st, err := os.Lstat(path)
+	if err != nil {
+		return nil, err
+	}
+	if !st.Mode().IsRegular() || st.Size() != pseudonymKeyLen || runtime.GOOS != "windows" && st.Mode().Perm()&0o077 != 0 {
+		return nil, fmt.Errorf("attest: %s must be a regular 32-byte file with mode 0600", path)
+	}
+	f, err := os.Open(path)
+	if err != nil {
+		return nil, err
+	}
+	defer func() { _ = f.Close() }()
+	if fs, serr := f.Stat(); serr != nil || !os.SameFile(st, fs) { // the file the checks judged is the file read
+		return nil, fmt.Errorf("attest: %s changed while it was read", path)
+	}
+	key := make([]byte, pseudonymKeyLen)
+	if _, err = io.ReadFull(f, key); err != nil {
+		return nil, err
+	}
+	return key, nil
+}
+
+// createPseudonymKey writes the key to a private temporary file and links it into place, so no reader
+// ever sees a partly written key file and a concurrent creator loses cleanly.
+func createPseudonymKey(path string) ([]byte, error) {
+	key := make([]byte, pseudonymKeyLen)
+	if _, err := rand.Read(key); err != nil {
+		return nil, err
+	}
+	tmp, err := os.CreateTemp(filepath.Dir(path), ".pseudonym-*")
+	if err != nil {
+		return nil, err
+	}
+	defer func() { _ = os.Remove(tmp.Name()) }()
+	if _, err = tmp.Write(key); err == nil {
+		err = tmp.Close()
+	} else {
+		_ = tmp.Close()
+	}
+	if err != nil {
+		return nil, err
+	}
+	if err = os.Link(tmp.Name(), path); errors.Is(err, os.ErrExist) {
+		return readPseudonymKey(path) // another process created it first
+	} else if err != nil {
+		return nil, err
+	}
+	return key, nil
+}
+
+// keyedPseudonym is the first 12 hex digits of HMAC-SHA256(key, msg).
+func keyedPseudonym(key []byte, msg string) string {
+	m := hmac.New(sha256.New, key)
+	_, _ = m.Write([]byte(msg))
+	return hex.EncodeToString(m.Sum(nil))[:12]
+}