diff --git a/.github/wiki/guides/ci/Trust-Security-and-Privacy.md b/.github/wiki/guides/ci/Trust-Security-and-Privacy.md index 9f0f99161..b07e9130e 100644 --- a/.github/wiki/guides/ci/Trust-Security-and-Privacy.md +++ b/.github/wiki/guides/ci/Trust-Security-and-Privacy.md @@ -149,7 +149,7 @@ A scope is offered only where it is enforced. The probe builds the same network ### Known limits - The exclusion set is the one this host can build for itself. A team coordinator that keeps the set in its own store can replace it with `run.RegisterNetscopeExclusions`; nothing registers one yet. -- The scheduler reads the proof for the local runner. A remote node's `network.enforced_scopes` is not reported yet, so no untrusted job is placed on a remote node on the strength of it. The agent does not yet pass the lease's network scope into the executor (container jobs there run with `none`), and `netscope.Effective`, which refuses `lan` and `privileged` for an untrusted job, is called by the scheduler's admission rules only. +- The scheduler reads the proof for the local runner. A remote node's `network.enforced_scopes` is not reported yet, so no untrusted job is placed on a remote node on the strength of it. A remote job's declared scope is checked against the class it was placed at three times: by the scheduler's admission rule, by the coordinator before it delivers the lease, and by the agent before it runs it. An untrusted job that declares `lan` or `privileged` fails there with E679 and never reaches the executor. A job that declares no scope runs as `none`, locally and on a node; the class default is not applied. - The proxy sees the destination host and port, not the encrypted content. It does not check the TLS server name, so a shared CDN host on the allowlist can be used for traffic to another site behind it, and uploading to an allowlisted registry is still an exfiltration path. `restricted` limits where a job can talk, not what it says. - Linux route discovery reads all routing tables through netlink, including policy routes. A public address that hairpins to a LAN service through the router cannot be learned from those tables. It still needs an observed egress address from the target network in the never-list; automatic public-NAT discovery is not implemented. - The sidecar bind-mounts a private staged copy of this process's binary with mode `0755`; the source binary's permissions are unchanged. The copy is removed with the attempt's network resources. Restricted and internet scopes still require Linux. @@ -475,7 +475,7 @@ nself ci serve admin audit checkpoint verify --pubkey [--js `export` (admin, TEAM coordinator only; PERSONAL mode refuses with a reason) seals the log, refuses a damaged chain, signs the head of the newest row with the coordinator's `ci-audit` key (`audit-checkpoint.key`, mode 0600, created on the first export in the data directory) and prints the key's sha256 fingerprint. Copy the checkpoint file and the fingerprint off this host: the key sits in the same data directory as the database, so only a checkpoint kept elsewhere detects a rewrite by someone who can write that directory. `verify` takes the fingerprint you pinned elsewhere; it never trusts the key in the data directory, refuses to run without `--pubkey`, checks the signature, re-verifies the whole chain and then compares the head of the checkpointed row with the live one. It exits 1 when the chain was rewritten (even with every hash recomputed) or rows were removed. Rows added after the checkpoint do not break it. -Handoff for later Tickets: the coordinator calls `run.UseAttestStore(st)` at start (the first attempt also sets it), a node revocation calls `run.RevokeNode`, and the coordinator that receives a result frame builds the signed-remote document with `producer.kind: signed-remote` and `signature: {kind: runner-ed25519, key_id, value}` from the frame; the agent signs the document it sends, so that document must already carry `attempt.id`, `job`, the real `repo_id`, `run_id`, `source_trust`, `policy_digest` and checks for its own job only (the agent's base document does not yet: D-1299; the marker file and the revoke caller are unwired: D-1300). +Handoff for later Tickets: the coordinator calls `run.UseAttestStore(st)` at start (the first attempt also sets it), a node revocation calls `run.RevokeNode`, and the coordinator that receives a result frame builds the signed-remote document with `producer.kind: signed-remote` and `signature: {kind: runner-ed25519, key_id, value}` from the frame; the agent signs the document it sends, so that document must already carry `attempt.id`, `job`, the real `repo_id`, `run_id`, `source_trust`, `policy_digest` and checks for its own job only. The lease carries those facts as its `evidence` context, read from the coordinator's record of the attempt, and the agent repeats them unchanged (the marker file and the revoke caller are unwired: D-1300). ## Policy sources and `nself ci policy` diff --git a/free/ci/cmd/agent/evidence_sign_e2e_test.go b/free/ci/cmd/agent/evidence_sign_e2e_test.go new file mode 100644 index 000000000..4fb40b1fd --- /dev/null +++ b/free/ci/cmd/agent/evidence_sign_e2e_test.go @@ -0,0 +1,158 @@ +package main + +import ( + "bytes" + "context" + "encoding/base64" + "encoding/json" + "errors" + "os" + "os/exec" + "path/filepath" + "testing" + + "github.com/nself-org/plugins/free/ci/internal/agent/nodekey" + "github.com/nself-org/plugins/free/ci/internal/agent/session" + "github.com/nself-org/plugins/free/ci/internal/attest" + "github.com/nself-org/plugins/free/ci/internal/model" + "github.com/nself-org/plugins/free/ci/internal/protocol" + cirun "github.com/nself-org/plugins/free/ci/internal/run" + "github.com/nself-org/plugins/free/ci/internal/store" +) + +// signedRig is a coordinator store with one signing-eligible node whose pinned key is the agent's node key, and +// a source tree at a real commit the attempts run against. +type signedRig struct { + *pullRig + key *nodekey.Key + repo string + head string +} + +func newSignedRig(t *testing.T) *signedRig { + t.Helper() + skipUnixOnly(t) + if _, err := exec.LookPath("git"); err != nil { + t.Skip("git not available") + } + state := useState(t) + r := &signedRig{pullRig: newPullRig(t), repo: t.TempDir()} + r.enroll(state, "n-1", "https://127.0.0.1:1") // the laptop fixture is signing eligible + var err error + if r.key, err = nodekey.LoadOrCreate(state); err != nil { + t.Fatal(err) + } + if err = r.st.PKINodeKeyPut(context.Background(), "n-1", r.key.KeyID(), r.key.PublicString()); err != nil { + t.Fatal(err) + } + git(t, r.repo, "init", "-q") + if err = os.WriteFile(filepath.Join(r.repo, "f.txt"), []byte("x"), 0o644); err != nil { + t.Fatal(err) + } + git(t, r.repo, "add", "f.txt") + git(t, r.repo, "commit", "-q", "-m", "a") + r.head = git(t, r.repo, "rev-parse", "HEAD") + return r +} + +// attempt records a running attempt of job "build" in project proj-1 at the rig's commit, leased to n-1. +func (r *signedRig) attempt(n string) string { + r.t.Helper() + ctx := context.Background() + p := store.PipelineRow{ID: "p" + n, Project: "proj-1", Revision: r.head, Trigger: "local", SourceTrust: "owner", PrivacyZone: "local-only", PolicyDigest: "policy-" + n, + InputDigest: "i", SelectionMode: "full", Status: "queued"} + for _, f := range []func() error{ + func() error { return r.st.CreatePipeline(ctx, p) }, + func() error { + return r.st.CreateJob(ctx, store.JobRow{ID: "j" + n, PipelineID: p.ID, Name: "build", Kind: "test", Idempotent: true, InfraMax: 1, InputDigest: "i"}) + }, + func() error { return r.st.CreateAttempt(ctx, store.AttemptRow{ID: "a" + n, JobID: "j" + n, N: 1}, "") }, + } { + if err := f(); err != nil { + r.t.Fatal(err) + } + } + if _, err := r.eng.Grant(ctx, "host", store.Demand{AttemptID: "a" + n, LeaseID: "l" + n, RunnerID: "n-1", CPU: 1, MemMB: 1, CapacityCPU: 100, CapacityMemMB: 100, TTL: r.leaseTTL()}, 1); err != nil { + r.t.Fatal(err) + } + return "a" + n +} + +// run runs the lease through the agent's lease path and returns the frame's evidence bytes (canonical JSON, as the +// session sends them) and the node's signature over them. +func (r *signedRig) run(l protocol.Lease) (frame []byte, keyID, sig string) { + r.t.Helper() + l.Checkout, l.Job = protocol.Checkout{Revision: r.head}, model.Job{Kind: "static", Run: []string{"true"}} + res, err := leaseRunner{coordinator: "coord"}.Run(context.Background(), session.JobRun{Lease: l, Setup: session.JobSetup{SourceDir: r.repo}}) + if err != nil { + r.t.Fatal(err) + } + dec := json.NewDecoder(bytes.NewReader(res.Evidence)) + dec.UseNumber() + var v any + if err = dec.Decode(&v); err != nil { + r.t.Fatal(err) + } + if frame, err = json.Marshal(v); err != nil { + r.t.Fatal(err) + } + s := evidenceSigner{key: r.key} + raw, err := s.Sign(frame) + if err != nil { + r.t.Fatal(err) + } + return frame, s.KeyID(), base64.RawStdEncoding.EncodeToString(raw) +} + +func (r *signedRig) verify(frame []byte, keyID, sig string) attest.RemoteReason { + err := attest.VerifyFrame(func() attest.Ledger { return store.AttestLedger{S: r.st} }, frame, keyID, sig) + if err == nil { + return "" + } + var re *attest.RemoteError + if !errors.As(err, &re) { + r.t.Fatalf("VerifyFrame = %v", err) + } + return re.Reason +} + +// D-1299 end to end: the coordinator puts the attempt's recorded facts into the lease (run.LeaseEvidence, the +// read the delivery path makes), the agent runs the lease against a real checkout and names them in its +// evidence, the node key signs the frame bytes, and the coordinator's frame check (attest.VerifyFrame with the +// real store ledger) accepts it. A lease without the context (the agent's own run id, repo unknown, source +// untrusted) and a context the node altered are both refused. +func TestSignedRemoteEvidenceVerifiesAgainstTheCoordinatorRecord(t *testing.T) { + r := newSignedRig(t) + ctx := context.Background() + + bare := r.attempt("1") + if reason := r.verify(r.run(protocol.Lease{AttemptID: bare, Trust: "owner"})); reason != attest.RemoteContextMismatch { + t.Fatalf("evidence without the lease's context: reason %q, want %s", reason, attest.RemoteContextMismatch) + } + + forged := r.attempt("2") + ev, err := cirun.LeaseEvidence(ctx, r.st, forged) + if err != nil || ev == nil { + t.Fatalf("LeaseEvidence = %v, %v", ev, err) + } + ev.PolicyDigest = "a-policy-the-node-prefers" + if reason := r.verify(r.run(protocol.Lease{AttemptID: forged, Trust: "owner", Evidence: ev})); reason != attest.RemoteContextMismatch { + t.Fatalf("evidence with an altered context: reason %q, want %s", reason, attest.RemoteContextMismatch) + } + + good := r.attempt("3") + if ev, err = cirun.LeaseEvidence(ctx, r.st, good); err != nil || ev == nil { + t.Fatalf("LeaseEvidence = %v, %v", ev, err) + } + frame, keyID, sig := r.run(protocol.Lease{AttemptID: good, Trust: "owner", Evidence: ev}) + if reason := r.verify(frame, keyID, sig); reason != "" { + t.Fatalf("signed evidence for the recorded attempt was refused: %s\n%s", reason, frame) + } + var doc model.Evidence + if err = json.Unmarshal(frame, &doc); err != nil { + t.Fatal(err) + } + if doc.Attempt == nil || doc.Attempt.ID != good || doc.Job == nil || *doc.Job != "build" || doc.Binding.State != "bound" || doc.RepoID != ev.RepoID { + t.Fatalf("the signed document does not name the attempt it ran: %s", frame) + } +} diff --git a/free/ci/cmd/agent/exec.go b/free/ci/cmd/agent/exec.go index 22f5ca639..faf9ef5ac 100644 --- a/free/ci/cmd/agent/exec.go +++ b/free/ci/cmd/agent/exec.go @@ -28,6 +28,8 @@ import ( "github.com/nself-org/plugins/free/ci/internal/evidence" ciexec "github.com/nself-org/plugins/free/ci/internal/exec" "github.com/nself-org/plugins/free/ci/internal/model" + "github.com/nself-org/plugins/free/ci/internal/netscope" + "github.com/nself-org/plugins/free/ci/internal/protocol" "github.com/nself-org/plugins/free/ci/internal/report" cirun "github.com/nself-org/plugins/free/ci/internal/run" ) @@ -38,14 +40,18 @@ func init() { // execSpec is the job a hosted workflow or a test hands to `exec`. type execSpec struct { - NetworkScope model.NetworkScope `json:"network_scope,omitempty"` - AttemptID string `json:"attempt_id"` - CoordinatorID string `json:"coordinator_id"` - JobID string `json:"job_id"` - Job model.Job `json:"job"` + NetworkScope model.NetworkScope `json:"network_scope,omitempty"` + // Trust is the class the coordinator admitted the job at (the lease's); with no class the job is untrusted. + Trust model.TrustClass `json:"trust,omitempty"` + AttemptID string `json:"attempt_id"` + CoordinatorID string `json:"coordinator_id"` + JobID string `json:"job_id"` + Job model.Job `json:"job"` // Revision is the full commit the job runs against (40 or 64 hex). Revision string `json:"revision"` SourceDir string `json:"source_dir,omitempty"` + // Evidence is the lease's evidence context: what the coordinator recorded for the attempt (D-1299). + Evidence *protocol.LeaseEvidence `json:"evidence,omitempty"` } // jobInputs are what a lease adds to its spec. @@ -172,6 +178,9 @@ func runSpec(ctx context.Context, spec execSpec, in jobInputs) (session.JobResul } else { unbindIfOtherRevision(ctx, spec, &done) } + if spec.Evidence != nil { // the builder names the checkout's own repo id; the signed document names the coordinator's + done.RepoID = base.RepoID + } } else { unbind(&done, "dirty", "the job ran without a source tree") } @@ -202,6 +211,12 @@ func execute(ctx context.Context, spec execSpec, in jobInputs) (ciexec.AttemptRe if err != nil { return ciexec.AttemptResult{Attempt: model.Attempt{ID: spec.AttemptID, State: "failed", FailureClass: "infra"}, ExitCode: -1, Started: now, Finished: time.Now(), Err: err}, err.Error(), nil } + // The lease's scope is checked again here, with the lease's class: the scheduler's admission rule is not the + // only thing between an untrusted job and the LAN (D-1331). An undeclared scope runs as none. + scope, err := netscope.RunScope(spec.Trust, spec.NetworkScope) + if err != nil { + return ciexec.AttemptResult{Attempt: model.Attempt{ID: spec.AttemptID, State: "failed", FailureClass: "infra"}, ExitCode: -1, Started: now, Finished: time.Now(), Err: err}, err.Error(), nil + } job.Env = map[string]string{} for k, v := range spec.Job.Env { job.Env[k] = v @@ -223,7 +238,7 @@ func execute(ctx context.Context, spec execSpec, in jobInputs) (ciexec.AttemptRe } raw := &tailBuffer{max: excerptMax} res := ex.Run(ctx, ciexec.JobSpec{AttemptID: spec.AttemptID, CoordinatorID: agentIncarnation(spec.CoordinatorID), Job: job, Command: job.Run, - Secrets: ex.Secrets, SourceDir: spec.SourceDir, Network: spec.NetworkScope, Mounts: in.Mounts, Home: home, Slot: in.Slot, Stdin: in.Stdin, Masks: in.Masks, RunnerRoot: in.RunnerRoot, Entrypoint: in.Entrypoint, + Secrets: ex.Secrets, SourceDir: spec.SourceDir, Network: scope, Mounts: in.Mounts, Home: home, Slot: in.Slot, Stdin: in.Stdin, Masks: in.Masks, RunnerRoot: in.RunnerRoot, Entrypoint: in.Entrypoint, CPU: jobCPU(in), MemoryMB: jobMemoryMB(in), BeforeJob: cacheBeforeJob(spec, job.Env), AfterJob: agentAfterJob(spec, in), Output: func(b []byte) { raw.Write(b) diff --git a/free/ci/cmd/agent/lease_network_test.go b/free/ci/cmd/agent/lease_network_test.go new file mode 100644 index 000000000..349c8d196 --- /dev/null +++ b/free/ci/cmd/agent/lease_network_test.go @@ -0,0 +1,78 @@ +package main + +import ( + "context" + "errors" + "fmt" + "strings" + "testing" + + "github.com/nself-org/plugins/free/ci/internal/agent/session" + ciexec "github.com/nself-org/plugins/free/ci/internal/exec" + "github.com/nself-org/plugins/free/ci/internal/invariants" + "github.com/nself-org/plugins/free/ci/internal/model" + "github.com/nself-org/plugins/free/ci/internal/protocol" + cirun "github.com/nself-org/plugins/free/ci/internal/run" + "github.com/nself-org/plugins/free/ci/internal/trust" +) + +// leaseScopeResult is what the executor saw for one (class, scope) lease: the scope it was handed, or that it +// was never called because the agent refused the lease first. +type leaseScopeResult struct { + ran bool + network model.NetworkScope + state string + excerpt string +} + +// runLeaseScope runs one container lease of class and declared scope through the agent's lease path, up to the +// isolator (which records the JobSpec and stops), and reports what happened. +func runLeaseScope(t *testing.T, class model.TrustClass, scope model.NetworkScope) leaseScopeResult { + t.Helper() + var got ciexec.JobSpec + cirun.RegisterIsolation(deliveredSpecObserver{&got, errors.New("stop before runtime")}) + cirun.RegisterImage(pinnedImageHook{}) + lease := protocol.Lease{AttemptID: "scope", Trust: class, NetworkScope: scope, Checkout: protocol.Checkout{Revision: rev40}, + Job: model.Job{Isolation: "container", Run: []string{"true"}}} + res, err := leaseRunner{coordinator: "test"}.Run(context.Background(), session.JobRun{Lease: lease}) + if err != nil { + t.Fatal(err) + } + return leaseScopeResult{ran: got.AttemptID != "", network: got.Network, state: res.Outcome.State, excerpt: string(res.Evidence)} +} + +// TestInvariant_I07 points I07 (untrusted never gets LAN) at the agent's lease path (D-1331): for every world of +// the invariant catalogue, a lease that declares lan or privileged for an untrusted job never reaches the +// executor (the attempt fails with E679), any other declared scope reaches it unchanged, and an undeclared scope +// runs as none. The worlds repeat (class, scope) pairs, so each pair runs once. +func TestInvariant_I07(t *testing.T) { + useState(t) + old := cirun.Registered + t.Cleanup(func() { cirun.Registered = old }) + seen := map[string]leaseScopeResult{} + invariants.Check(t, func(w invariants.World) error { + key := string(w.Trust) + "/" + string(w.Network) + r, ok := seen[key] + if !ok { + r = runLeaseScope(t, w.Trust, w.Network) + seen[key] = r + } + untrusted := trust.DecodeTrust(string(w.Trust)) == "untrusted" + lan := w.Network == "lan" || w.Network == "privileged" + switch { + case untrusted && lan && r.ran: + return fmt.Errorf("an untrusted lease declaring %s reached the executor with %q", w.Network, r.network) + case untrusted && lan && (r.state != "failed" || !strings.Contains(r.excerpt, "E679")): + return fmt.Errorf("an untrusted lease declaring %s ended %q without E679", w.Network, r.state) + case untrusted && lan: + return nil + case !r.ran: + return fmt.Errorf("a %s lease declaring %q never reached the executor", w.Trust, w.Network) + case w.Network == "" && r.network != "none": + return fmt.Errorf("an undeclared scope ran as %q, want none", r.network) + case w.Network != "" && r.network != w.Network: + return fmt.Errorf("a %s lease declaring %s ran as %q", w.Trust, w.Network, r.network) + } + return nil + }) +} diff --git a/free/ci/cmd/agent/main.go b/free/ci/cmd/agent/main.go index d9a25aea8..f59d4272a 100644 --- a/free/ci/cmd/agent/main.go +++ b/free/ci/cmd/agent/main.go @@ -102,19 +102,27 @@ func jobOutcome(spec execSpec, res ciexec.AttemptResult, tail string) cirun.JobO // baseEvidence is the run document before the P7-CI-38 builder completes it: // one check for the one job, assembled by the run package so the check and // verdict logic exists once, with the producer marked self-reported until the -// result signer (P7-TRUST-20) signs it. +// result signer (P7-TRUST-20) signs it. A lease's evidence context (D-1299) +// supplies the run, repository, source trust, policy digest and selection mode +// the coordinator recorded, so the signed document verifies; without one the +// agent claims nothing (a run id of its own, repo unknown, source untrusted). func baseEvidence(spec execSpec, o cirun.JobOutcome, start, end time.Time) model.Evidence { id := sha256.Sum256([]byte(spec.CoordinatorID + "/" + spec.AttemptID)) - runID := hex.EncodeToString(id[:8]) - return cirun.SingleJobEvidence(cirun.SingleJobInput{ - RunID: runID, Revision: spec.Revision, RepoID: "unknown", + runID, repoID, source, mode, policy := hex.EncodeToString(id[:8]), "unknown", "untrusted", model.SelectionMode("full"), "" + if c := spec.Evidence; c != nil { + runID, repoID, source, mode, policy = c.RunID, c.RepoID, c.SourceTrust, model.SelectionMode(c.SelectionMode), c.PolicyDigest + } + ev := cirun.SingleJobEvidence(cirun.SingleJobInput{ + RunID: runID, Revision: spec.Revision, RepoID: repoID, Pipeline: model.PipelineRef{ID: runID, Name: spec.JobID, Kind: "adhoc"}, Trigger: "api", - Source: "untrusted", // the coordinator classifies the source; the agent claims nothing - Selection: model.Selection{Mode: "full", Reason: "selection.full_required"}, + Source: source, + Selection: model.Selection{Mode: mode, Reason: "selection.full_required"}, Runner: model.RunnerRef{ID: "agent", Provider: "agent", Platform: runtime.GOOS + "/" + runtime.GOARCH, Isolation: "process"}, Producer: model.Producer{Kind: "self-reported", Version: "1"}, Job: o, Start: start, End: end, }) + ev.PolicyDigest = policy + return ev } // unbindIfOtherRevision unbinds evidence whose named revision is not the commit diff --git a/free/ci/cmd/agent/serve.go b/free/ci/cmd/agent/serve.go index 94f4ccd93..89b643a65 100644 --- a/free/ci/cmd/agent/serve.go +++ b/free/ci/cmd/agent/serve.go @@ -147,7 +147,10 @@ func (r leaseRunner) Run(ctx context.Context, j session.JobRun) (session.JobResu // run on an empty tree would report a pass for code it never saw. return session.JobResult{}, errors.New("the lease has a checkout but no hook staged it") } - spec := execSpec{AttemptID: l.AttemptID, CoordinatorID: r.coordinator, JobID: "job", Job: l.Job, NetworkScope: l.NetworkScope, Revision: l.Checkout.Revision, SourceDir: j.Setup.SourceDir} + spec := execSpec{AttemptID: l.AttemptID, CoordinatorID: r.coordinator, JobID: "job", Job: l.Job, NetworkScope: l.NetworkScope, Trust: l.Trust, Revision: l.Checkout.Revision, SourceDir: j.Setup.SourceDir, Evidence: l.Evidence} + if l.Evidence != nil && l.Evidence.Job != "" { // the evidence names the job the coordinator recorded for the attempt + spec.JobID = l.Evidence.Job + } in := jobInputs{Slot: j.Slot, Env: j.Setup.Env, Secrets: j.Setup.Secrets, Mounts: j.Setup.Mounts, Output: j.Output, Tail: j.Tail, Skip: j.Skip} if len(j.Credential) > 0 { if err := wrapCredential(&spec, &in, j); err != nil { diff --git a/free/ci/internal/dispatch/deliver_test.go b/free/ci/internal/dispatch/deliver_test.go index 6ec19ae63..427a4796b 100644 --- a/free/ci/internal/dispatch/deliver_test.go +++ b/free/ci/internal/dispatch/deliver_test.go @@ -238,7 +238,8 @@ func TestDispatchLeaseWithoutSecretRefsForAnIneligibleNode(t *testing.T) { if _, err := d.Submit(context.Background(), secretJob(t, id)); err != nil { t.Fatal(err) } - if got := wait(t, remote.got).lease; len(got.Secrets) != 0 || got.Trust != "" { + // The lease still carries the class the job was placed at: the node checks the network scope by it (D-1331). + if got := wait(t, remote.got).lease; len(got.Secrets) != 0 || got.Trust != secretJob(t, id).Sched.Trust { t.Fatalf("an ineligible node got secrets %+v trust %q", got.Secrets, got.Trust) } } diff --git a/free/ci/internal/netscope/runscope_test.go b/free/ci/internal/netscope/runscope_test.go new file mode 100644 index 000000000..15bf0743f --- /dev/null +++ b/free/ci/internal/netscope/runscope_test.go @@ -0,0 +1,40 @@ +package netscope + +import ( + "errors" + "testing" + + "github.com/nself-org/plugins/free/ci/internal/model" +) + +// RunScope (D-1331): an undeclared scope runs as none for every class (the class default never widens a job that +// declared nothing), and a declared scope gets the same refusal Effective makes. +func TestRunScope(t *testing.T) { + for _, class := range []model.TrustClass{"owner", "internal", "collaborator", "untrusted", "", "bogus"} { + if got, err := RunScope(class, ""); err != nil || got != "none" { + t.Fatalf("RunScope(%q, \"\") = %q, %v; want none", class, got, err) + } + } + for _, c := range []struct { + class model.TrustClass + scope model.NetworkScope + refused bool + }{ + {"untrusted", "lan", true}, {"untrusted", "privileged", true}, {"", "lan", true}, {"bogus", "privileged", true}, + {"owner", "nowhere", true}, + {"untrusted", "restricted", false}, {"untrusted", "internet", false}, {"untrusted", "none", false}, + {"owner", "lan", false}, {"internal", "privileged", false}, {"collaborator", "lan", false}, + } { + got, err := RunScope(c.class, c.scope) + var ne *Error + if c.refused { + if !errors.As(err, &ne) || ne.Code != CodeScopeFailed || got != "" { + t.Fatalf("RunScope(%q, %q) = %q, %v; want E679", c.class, c.scope, got, err) + } + continue + } + if err != nil || got != c.scope { + t.Fatalf("RunScope(%q, %q) = %q, %v", c.class, c.scope, got, err) + } + } +} diff --git a/free/ci/internal/netscope/scope.go b/free/ci/internal/netscope/scope.go index f4db4d2f6..b4a753592 100644 --- a/free/ci/internal/netscope/scope.go +++ b/free/ci/internal/netscope/scope.go @@ -28,6 +28,18 @@ func Effective(class model.TrustClass, declared model.NetworkScope) (model.Netwo return declared, nil } +// RunScope returns the scope an attempt of class runs under on a path that hands the executor its scope (a +// remote lease, the agent's exec spec): the same refusal Effective makes for a declared scope, and none (the +// narrowest scope, what the executor runs an empty scope as) for an undeclared one, so the class default never +// widens a job that declared nothing. The coordinator calls it before it delivers a lease and the agent again +// before it runs one (D-1331). +func RunScope(class model.TrustClass, declared model.NetworkScope) (model.NetworkScope, error) { + if declared == "" { + return "none", nil + } + return Effective(class, declared) +} + // Enforceable lists the scopes this package can ever enforce on a runner: lan and privileged are raw network // access and need no enforcement, so they are not in the list. func Enforceable() []model.NetworkScope { diff --git a/free/ci/internal/protocol/messages.go b/free/ci/internal/protocol/messages.go index 1810ae052..281786ee5 100644 --- a/free/ci/internal/protocol/messages.go +++ b/free/ci/internal/protocol/messages.go @@ -118,10 +118,23 @@ type Lease struct { Job model.Job `json:"job"` Checkout Checkout `json:"checkout"` Secrets []SecretRef `json:"secrets"` - Trust model.TrustClass `json:"trust,omitempty"` // the class the job was admitted at; set with Secrets, so a node judges its own eligibility by it + Trust model.TrustClass `json:"trust,omitempty"` // the class the job was admitted at; a node judges its own secret eligibility and the job's network scope by it ExpiresAt string `json:"expires_at"` Cache *LeaseCache `json:"cache,omitempty"` RunnerCredential *RunnerCredential `json:"runner_credential,omitempty"` + Evidence *LeaseEvidence `json:"evidence,omitempty"` +} + +// LeaseEvidence is what the coordinator recorded for the attempt (its run, repository, source trust, policy +// digest, selection mode and job name). The node's evidence names exactly these, so a signed result verifies +// against the coordinator's record (attest.RunnerVerifier) without a field changing after it is signed (D-1299). +type LeaseEvidence struct { + RunID string `json:"run_id"` + RepoID string `json:"repo_id"` + SourceTrust string `json:"source_trust"` + PolicyDigest string `json:"policy_digest"` + SelectionMode string `json:"selection_mode"` + Job string `json:"job"` } type Chunk struct { Stream string `json:"stream"` diff --git a/free/ci/internal/run/dispatch_hook.go b/free/ci/internal/run/dispatch_hook.go index 8a27d1cf9..9a1cff712 100644 --- a/free/ci/internal/run/dispatch_hook.go +++ b/free/ci/internal/run/dispatch_hook.go @@ -23,6 +23,7 @@ import ( compat "github.com/nself-org/cli/sdk/go/v2/compat" "github.com/nself-org/plugins/free/ci/internal/dispatch" ciexec "github.com/nself-org/plugins/free/ci/internal/exec" + "github.com/nself-org/plugins/free/ci/internal/netscope" "github.com/nself-org/plugins/free/ci/internal/protocol" "github.com/nself-org/plugins/free/ci/internal/sched" "github.com/nself-org/plugins/free/ci/internal/schedpolicy" @@ -197,13 +198,18 @@ type carriers interface { } func newTransportAdapter(sink transport.Sink, reg carriers, root string) dispatch.Transport { - return transportAdapter{sink: sink, reg: reg, root: root} + a := transportAdapter{sink: sink, reg: reg, root: root} + if rs, ok := sink.(*remoteSink); ok && rs != nil { + a.st = rs.st + } + return a } type transportAdapter struct { sink transport.Sink reg carriers - root string // the repository whose revision a node receives + root string // the repository whose revision a node receives + st *store.Store // the run's store: the attempt facts the lease's evidence context repeats (nil: none) } func (transportAdapter) Known(kind string) bool { @@ -220,15 +226,21 @@ func (a transportAdapter) Deliver(ctx context.Context, d dispatch.Delivery) erro if RemoteRepo != nil && a.root != "" { RemoteRepo(a.root) } + // The scheduler's admission rule is not the only gate: an untrusted job that declares lan or privileged is + // refused here too, and the node checks the lease again with the class it carries (D-1331). + if _, err := netscope.RunScope(d.Trust, d.Network); err != nil { + return err + } refs, err := leaseSecretRefs(d) if err != nil { return err } - lease := protocol.Lease{NetworkScope: d.Network, LeaseID: d.LeaseID, AttemptID: d.AttemptID, Epoch: uint64(d.Epoch), Token: d.Token, CompatMode: protocol.Compat15, Job: d.Job, - Checkout: protocol.Checkout{Revision: d.Revision}, Secrets: refs, ExpiresAt: d.ExpiresAt.UTC().Format(time.RFC3339)} - if len(refs) > 0 { - lease.Trust = d.Trust + evidence, err := LeaseEvidence(ctx, a.st, d.AttemptID) + if err != nil { + return err } + lease := protocol.Lease{Evidence: evidence, NetworkScope: d.Network, LeaseID: d.LeaseID, AttemptID: d.AttemptID, Epoch: uint64(d.Epoch), Token: d.Token, CompatMode: protocol.Compat15, Job: d.Job, + Checkout: protocol.Checkout{Revision: d.Revision}, Secrets: refs, Trust: d.Trust, ExpiresAt: d.ExpiresAt.UTC().Format(time.RFC3339)} sink := bindLeaseSink(a.sink, leaseFacts{AttemptID: d.AttemptID, LeaseID: d.LeaseID, Epoch: d.Epoch, Job: d.Job, Hosted: d.Hosted}) sink = prepareCache(ctx, sink, &lease, a.root, d.Node.Identity.ID, d.Hosted, remoteCachePlatform(d.Node), d.Node.Identity.Transport) return transport.Deliver(ctx, d.Node, lease, sink) diff --git a/free/ci/internal/run/dispatch_netscope_test.go b/free/ci/internal/run/dispatch_netscope_test.go new file mode 100644 index 000000000..2fad9c965 --- /dev/null +++ b/free/ci/internal/run/dispatch_netscope_test.go @@ -0,0 +1,67 @@ +package run + +import ( + "context" + "errors" + "fmt" + "testing" + "time" + + "github.com/nself-org/plugins/free/ci/internal/model" + "github.com/nself-org/plugins/free/ci/internal/netscope" + "github.com/nself-org/plugins/free/ci/internal/transport" +) + +// D-1331: the coordinator refuses to deliver a lease whose declared scope the job's class may not have (the +// scheduler's admission rule is not the only gate), and every lease carries the class the job was placed at, +// with or without secrets, so the node can check the scope again. +func TestDeliverRefusesALANScopeForAnUntrustedJobAndCarriesTheClass(t *testing.T) { + probe := &probeDriver{kind: fmt.Sprintf("netscopeprobe%d", time.Now().UnixNano())} + if err := transport.Register(probe); err != nil { + t.Fatal(err) + } + var n model.Capability + n.Identity.ID, n.Identity.Transport = "x1", probe.kind + job := model.Job{Kind: "test", Run: []string{"true"}} + for _, c := range []struct { + trust model.TrustClass + scope model.NetworkScope + refused bool + }{ + {"untrusted", "lan", true}, + {"untrusted", "privileged", true}, + {"", "lan", true}, // no class reads as untrusted + {"untrusted", "internet", false}, + {"untrusted", "", false}, + {"owner", "lan", false}, + {"collaborator", "privileged", false}, + } { + t.Run(fmt.Sprintf("%s/%s", c.trust, c.scope), func(t *testing.T) { + probe.mu.Lock() + probe.lease.AttemptID = "" + probe.mu.Unlock() + d := deliveryFor(n, job) + d.Trust, d.Network = c.trust, c.scope + err := newTransportAdapter(nil, nil, "").Deliver(context.Background(), d) + probe.mu.Lock() + got := probe.lease + probe.mu.Unlock() + if c.refused { + var ne *netscope.Error + if !errors.As(err, &ne) || ne.Code != netscope.CodeScopeFailed { + t.Fatalf("Deliver = %v, want E679", err) + } + if got.AttemptID != "" { + t.Fatalf("a refused lease reached the node: %+v", got) + } + return + } + if err != nil { + t.Fatal(err) + } + if got.NetworkScope != c.scope || got.Trust != c.trust { + t.Fatalf("the lease carries scope=%q trust=%q, want %q %q", got.NetworkScope, got.Trust, c.scope, c.trust) + } + }) + } +} diff --git a/free/ci/internal/run/lease_evidence.go b/free/ci/internal/run/lease_evidence.go new file mode 100644 index 000000000..053dc8714 --- /dev/null +++ b/free/ci/internal/run/lease_evidence.go @@ -0,0 +1,30 @@ +package run + +// Purpose: the evidence context a remote lease carries (D-1299): what the coordinator recorded for the +// attempt, read with the very query the signed-evidence verifier compares with (store.AttestLedger), so a +// node that names it in its evidence and signs that evidence produces a document attest.RunnerVerifier +// accepts. +// Inputs: the run's store and the attempt id of the lease being delivered. +// Outputs: a *protocol.LeaseEvidence, or nil when there is no store or no recorded lease holder. +// Constraints: a store read error fails the delivery (the lease would otherwise go out with evidence that +// cannot verify); the node never decides these fields, it only repeats them. + +import ( + "context" + + "github.com/nself-org/plugins/free/ci/internal/protocol" + "github.com/nself-org/plugins/free/ci/internal/store" +) + +// LeaseEvidence reads the attempt's recorded facts for its lease (exported for the agent's end-to-end signing test). +func LeaseEvidence(ctx context.Context, st *store.Store, attemptID string) (*protocol.LeaseEvidence, error) { + if st == nil { + return nil, nil + } + f, ok, err := store.AttestLedger{S: st}.AttemptFacts(ctx, attemptID) + if err != nil || !ok { + return nil, err + } + return &protocol.LeaseEvidence{RunID: f.RunID, RepoID: f.RepoID, SourceTrust: f.SourceTrust, PolicyDigest: f.PolicyDigest, + SelectionMode: f.SelectionMode, Job: f.Job}, nil +} diff --git a/free/ci/internal/run/lease_evidence_test.go b/free/ci/internal/run/lease_evidence_test.go new file mode 100644 index 000000000..b360658b8 --- /dev/null +++ b/free/ci/internal/run/lease_evidence_test.go @@ -0,0 +1,75 @@ +package run + +import ( + "context" + "crypto/sha256" + "encoding/hex" + "fmt" + "path/filepath" + "testing" + "time" + + "github.com/nself-org/plugins/free/ci/internal/lease" + "github.com/nself-org/plugins/free/ci/internal/model" + "github.com/nself-org/plugins/free/ci/internal/protocol" + "github.com/nself-org/plugins/free/ci/internal/store" + "github.com/nself-org/plugins/free/ci/internal/transport" +) + +// D-1299: a lease the run's adapter delivers carries the attempt's recorded facts (run, repository id derived +// as the verifier derives it, source trust, policy digest, selection mode, job name), so the node can name them +// in the evidence it signs. An attempt with no recorded lease holder gets no context. +func TestDeliveredLeaseCarriesTheAttemptsEvidenceContext(t *testing.T) { + ctx := context.Background() + st, err := store.Open(filepath.Join(t.TempDir(), "state.db"), store.Options{}) + if err != nil { + t.Fatal(err) + } + t.Cleanup(func() { _ = st.Close() }) + p := store.PipelineRow{ID: "p1", Project: "acme/widgets", Revision: "0123456789abcdef0123456789abcdef01234567", Trigger: "local", SourceTrust: "collaborator", + PrivacyZone: "local-only", PolicyDigest: "policy-1", InputDigest: "i", SelectionMode: "affected", Status: "queued"} + for _, f := range []func() error{ + func() error { return st.CreatePipeline(ctx, p) }, + func() error { + return st.CreateJob(ctx, store.JobRow{ID: "j1", PipelineID: "p1", Name: "unit", Kind: "test", Idempotent: true, InfraMax: 1, InputDigest: "i"}) + }, + func() error { return st.CreateAttempt(ctx, store.AttemptRow{ID: "a1", JobID: "j1", N: 1}, "") }, + func() error { + return st.CreateJob(ctx, store.JobRow{ID: "j2", PipelineID: "p1", Name: "lint", Kind: "test", Idempotent: true, InfraMax: 1, InputDigest: "i"}) + }, + func() error { return st.CreateAttempt(ctx, store.AttemptRow{ID: "a2", JobID: "j2", N: 1}, "") }, + } { + if err := f(); err != nil { + t.Fatal(err) + } + } + if _, err := lease.New(st, nil, lease.Config{}).Grant(ctx, "host", store.Demand{AttemptID: "a1", LeaseID: "lease-a1", RunnerID: "x1", CPU: 1, MemMB: 1, + CapacityCPU: 100, CapacityMemMB: 100, TTL: time.Minute}, 1); err != nil { + t.Fatal(err) + } + probe := &probeDriver{kind: fmt.Sprintf("evidenceprobe%d", time.Now().UnixNano())} + if err := transport.Register(probe); err != nil { + t.Fatal(err) + } + var n model.Capability + n.Identity.ID, n.Identity.Transport = "x1", probe.kind + adapter := newTransportAdapter(newRemoteSink(st), nil, "") + sum := sha256.Sum256([]byte("acme/widgets")) + want := protocol.LeaseEvidence{RunID: "p1", RepoID: hex.EncodeToString(sum[:8]), SourceTrust: "collaborator", PolicyDigest: "policy-1", SelectionMode: "affected", Job: "unit"} + for _, c := range []struct { + attempt string + want *protocol.LeaseEvidence + }{{"a1", &want}, {"a2", nil}} { + d := deliveryFor(n, model.Job{Kind: "test", Run: []string{"true"}}) + d.AttemptID, d.Trust = c.attempt, "collaborator" + if err := adapter.Deliver(ctx, d); err != nil { + t.Fatal(err) + } + probe.mu.Lock() + got := probe.lease.Evidence + probe.mu.Unlock() + if (got == nil) != (c.want == nil) || got != nil && *got != *c.want { + t.Fatalf("attempt %s: lease evidence %+v, want %+v", c.attempt, got, c.want) + } + } +} diff --git a/free/ci/internal/run/remote_await_test.go b/free/ci/internal/run/remote_await_test.go index 39df9ea80..2268defbb 100644 --- a/free/ci/internal/run/remote_await_test.go +++ b/free/ci/internal/run/remote_await_test.go @@ -473,7 +473,7 @@ func TestLeaseSecretRefsGoOnlyToAnEligibleNodeThatPassesTheGate(t *testing.T) { if len(got.Secrets) != c.wantRefs { t.Fatalf("the lease carries %d refs %v, want %d", len(got.Secrets), got.Secrets, c.wantRefs) } - if (c.wantRefs > 0) != (got.Trust != "") || c.wantRefs > 0 && got.Trust != c.trust { + if got.Trust != c.trust { // every lease carries its class, with or without refs (D-1331: the node checks the scope by it) t.Fatalf("lease trust = %q with %d refs", got.Trust, c.wantRefs) } for _, r := range got.Secrets { diff --git a/free/ci/schemas/runner-protocol.v1.schema.json b/free/ci/schemas/runner-protocol.v1.schema.json index 889bc5c68..586a52a56 100644 --- a/free/ci/schemas/runner-protocol.v1.schema.json +++ b/free/ci/schemas/runner-protocol.v1.schema.json @@ -663,6 +663,45 @@ "minimum": 1, "type": "integer" }, + "evidence": { + "anyOf": [ + { + "additionalProperties": false, + "properties": { + "job": { + "type": "string" + }, + "policy_digest": { + "type": "string" + }, + "repo_id": { + "type": "string" + }, + "run_id": { + "type": "string" + }, + "selection_mode": { + "type": "string" + }, + "source_trust": { + "type": "string" + } + }, + "required": [ + "run_id", + "repo_id", + "source_trust", + "policy_digest", + "selection_mode", + "job" + ], + "type": "object" + }, + { + "type": "null" + } + ] + }, "expires_at": { "type": "string" },